thbar

thbar

Resumable and/or accelerated downloads with Elixir?

Hi!

A part of my ETL/data-pipelining work involves dealing with input files whose size can reach 1 to 2 GB in size (compressed).

When working with Ruby, I usually shell out to tools like axel to handle the download in a way that can be resumed, should a disconnection occur, or even accelerated, if the source server is doing some throttling in a way or another.

To your knowledge, is there an already built solution to achieve that in Elixir or Erlang?

Solutions I’m considering:

  • Shelling out to Axel using Porcelain (but porcelain has issues apparently)
  • Implementing a resumable download manager myself on top of an existing Elixir HTTP library
  • Wrapping a Rust download manager such as zou with e.g. rustler

If anyone dived into this topic, your feedback is most welcome!

– Thibaut

Most Liked

idi527

idi527

For resumable downloads, I’d think it could be done with most http clients with a bit of work. For example, with httpoison, you’d write the stream of bytes (stream_to: self() option) to a temporary file and only “save” it (move the temporary file to a “permanent” location similarly to how Plug.Upload works) once the download is complete. If there is already a temporary file when a download process starts, it would read it, calculate the “offset” needed for the range header, and only download what it needs to.

For accelerated downloads, you’d need to split the file into chunks of some configurable size and download each with the “resumable” downloader described above, then cat them into a single file at the final destination …

This is what I’d try before reaching out to rust or porcelain.

Zesky665

Zesky665

You can just use a Task to run a cli app(using System.cmd\3) to do the same thing.

Where Next?

Popular in Questions Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
Kagamiiiii
Student & New to elixir. Nice language. I want to convert a english character, e.g. “a”, which is stored in a variable, to it’s asci...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
kostonstyle
Hi all I want to have a unix time, from the current time plus 1 hour. DateTime.now + 1 hour How to get it in elixir? Thanks
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New

Other popular topics Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement