jc00ke
Streaming from Ecto, through CSV, Zip, then up to S3... w/o going through a temp file
Hi there,
I’ve got a data export that has the following flow:
I’m using Ecto, CSV.encode/2, Zstream.zip/2 and ExAws.S3.Upload.stream_file/2.
I tried at first to avoid the temp file (I’m on Heroku, so not entirely ideal to use the filesystem, though passable) but I wasn’t able to get anything working. I tried writing my own function that used upload_chunk!/3 but it seems the first entry in the zip stream is not a binary.
Thanks for any guidance or advice you can spare!
Most Liked
benwilson512
Pretty sure you can pipe the gzip stream directly into: https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#upload/4
The examples use https://hexdocs.pm/ex_aws_s3/ExAws.S3.Upload.html#stream_file/2 to create the stream but all that does is a regular File.stream! with the minimum chunk size set (5mb) https://github.com/ex-aws/ex_aws_s3/blob/v2.0.2/lib/ex_aws/s3/upload.ex#L53.
Do note that your chunks need to be at least 5mb which is the minimum multi part upload size for s3. If the compressed file is less than that you should just do a regular ExAws.S3.put_object call.
LostKobrakai
dimitarvp
If you make a GitHub repo and are willing to provide sample data (anonymised / mocked) and an access to S3[-compatible] storage then I’d gladly pair with you to find a solution.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
We build reliable cloud platforms for business-critical systems.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"