Stream a remote file to S3
While attempting to work with my gif collection, I was experimenting with how to capture gifs from the internet and place them into a S3 bucket for later use. I found that it was possible to stream a remote file directly to S3.
require 'aws-sdk' require 'open-uri' require 'sequel' require 'digest/md5' class RemoteFile def initialize(url) @url = url end def s3 @s3 ||= Aws::S3::Resource.new(region: ENV['AWS_REGION']) end def md5_hash Digest::MD5.hexdigest(@url) end def obj(bucket_name) s3.bucket(bucket_name).object(md5_hash) end def url URI.parse(URI.escape(@url)) end def upload_to_s3(bucket_name:) open(url) do |file| obj(bucket_name).put(body: file) end end end RemoteFile.new('https://i.imgur.com/DO3Hr4A.gif') .upload_to_s3(bucket_name: 'mah_gifs')
This code snippet assumes you have