Hey, Waffle author here.

You can do this via the Waffle library like this:

Avatar.store("http://example.com/file.png") #=> {:ok, "file.png"}

You only need to configure access for your S3 storage and create an uploader.

Please, feel free to ask if you’ll have any troubles.

I’ll essentially be using your prefered http client library to fetch the files and using your prefered aws library to push the data to S3. From there you could layer optimizations like streaming or parallelism on top if needed.

I have used Waffle for this.

https://hexdocs.pm/waffle_ecto/Waffle.Ecto.Schema.html#cast_attachments/4-examples

s3cmd and s3 have a sync command that can synchronize between buckets and regions but I don’t know if you can use it to sync between s3 compatible providers. It seems like a logical goal but configuring multiple providers isn’t something I’ve done to know if it’s possible.

GitHub - s3tools/s3cmd: Official s3cmd repo -- Command line tool for managing Amazon S3 and CloudFront services or sync — AWS CLI 1.27.145 Command Reference lists the aws cli. I’m not at my laptop atm to look at s3cmd’s output but it’s one of the more robust CLIs that seemingly covers everything I throw at it.

My coworker is in the process of synchronizing our entire bucket between local and another provider due to issues with digital ocean spaces. I’ll try to hit him up later today to see how he’s getting that done. It wouldn’t be Elixir but knowing his steps could be helpful to work backwards if you go down the http client route.