ryanwinchester
Streaming from S3 to client with constant memory?
Greetings fellow humans,
First, I’ll tell you why I’m not using pre-signed URLs: because this is for sensitive files, that I want to control access to at a much more granular level than signed URLs allow.
I’ve been looking at aws-elixir and ex_aws_s3.
For now, I’m planning to use cloud functions, and want to keep memory limits as low as possible.
So, my objective is to stream a file from S3, to the client, while keeping memory usage constant and not running out.
It looks like with ex_aws_s3 I can do:
ExAws.S3.download_file(..., :memory)
|> ExAws.stream!()
|> Enum.reduce_while(Plug.Conn.send_chunked(), &()...)
but I’m worried about not being able to control the memory usage it could be a problem?
Another option @philss mentioned in aws-elixir in this issue is using a range option, which I guess would give more control.
What do you fine elixir folks think I should do?
Marked As Solved
philss
Cool! I made a small PoC using aws-elixir and Finch that kept the memory usage even lower (ends in 4m05). It’s not a fair comparison because it’s not streaming to the client.
Here is the code: aws-s3-stream-download-poc/lib/download_manager.ex at main · philss/aws-s3-stream-download-poc · GitHub (there is a custom AWS.HTTPClient implementation using Finch inside the project).
Also Liked
alexandre
Some months ago I wrote a stream downloader with Mint.
I had the opposite problem: stream from clients to S3. I needed to control the chunk size since S3 limits to 5MB.
That was accomplished with:
url
|> Downloader.stream_body!()
|> Downloader.chunk_bytes(5_000_000)
|> ExAws.S3.upload(s3_bucket, filename, opts)
|> ExAws.request!()
josevalim
The concurrency of 8 for download_file is going to increase the memory size (and likely the amount of data references). If the goal is to stream, I would try to go with Finch (and/or Mint) + Range queries.
philss
I’m getting a
403, and can’t tell if I’m using the function arguments wrong or if I’m deriving the key incorrectly, so I don’t know where to waste my time
The usage is correct. The problem was a bug in the signature algorithm, which was not related to SSE-C 
I managed to fix it on master: Fix signature bug when headers are similar · aws-beam/aws-elixir@306fe41 · GitHub
This week we may release a new version of “aws-elixir” to Hex.pm.
evadne
You can use a pre-signed URL with a signature that has an insanely short period of validity. Sign the URL every time. Should be much simpler. Having traffic go through your host can create a massive headache when the time comes to pay your bills!
evadne
TBF…
ExAws.S3.download_file/4: Defaults to a concurrency of 8, chunk size of 1MB, and a timeout of 1 minute.
So you should be ok. The stream isn’t started until the customer starts downloading anyway.
If you want to avoid the server-side round-trip, then you will have to expose the headers and start the download on the client with some JavaScript. You can use XMLHttpRequest to arrange the headers and get a blob back, then make a Data URI out of the blob, and serve it to the user… this works as long as the file is not large.
Can try GitHub - eligrey/FileSaver.js: An HTML5 saveAs() FileSaver implementation for the saving part too
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"