Are you using ExAws.S3 ?

In the past I did a service like wetransfer, using S3. So, if you want to use your application just for authorisation you can use a presigned url.

client --> (your backend generate a presigned url) —> redirect the client to this url —> client downloads

https://hexdocs.pm/ex_aws_s3/ExAws.S3.html#presigned_url/5

In this way the client downloads directly from S3, without using bandwidth of your servers.

I had the time to write down an idea of wrapper around HTTPoison, to make it an Elixir stream.

defmodule HTTPDownload do

  def stream!(url) do
    Stream.resource(
      fn -> start_request(url) end,
      fn ref -> 
        case receive_response(ref) do
          #returning the chunk to the stream
          {:ok, {:chunk, chunk}} -> 
            HTTPoison.stream_next(ref)
            {[chunk], ref}
          {:ok, msg} -> 
            IO.inspect(msg)
            HTTPoison.stream_next(ref)
            {[], ref}
          {:error, error} -> 
            IO.puts("ERROR")
            raise("error #{inspect error}")
          :done -> {:halt, ref}
        end
      end,
      fn ref -> :hackney.stop_async(ref) end
    )
  end


  defp start_request(url) do
    {:ok, ref} = HTTPoison.get(url, %{}, stream_to: self(), async: :once)
    ref
  end

  defp receive_response(ref) do
    id = ref.id
    receive do 
      %HTTPoison.AsyncStatus{code: code, id: ^id} when 200 <= code and code < 300 -> 
        {:ok, {:status_code, code}}
      %HTTPoison.AsyncStatus{code: code, id: ^id} ->
        {:error, {:status_code, code}}

      %HTTPoison.AsyncHeaders{headers: headers, id: ^id}->
        {:ok, {:headers, headers}}
      
      %HTTPoison.AsyncChunk{chunk: chunk, id: ^id}->
        {:ok, {:chunk, chunk}}

      %HTTPoison.AsyncEnd{id: ^id}-> :done
    end
  end
end

So in this way you can get a stream from a http response and you can use it like this:

HTTPDownloader.stream!( url_to_my_s3_file )
|> Enum.each(fn chunk-> send_chunk_to_client(client_conn, chunk) end)

So, if you don’t want to deal yourself with AWS HTTP API, you can get the presigned url using the ex_aws_s3 library, and use the url in your backend to get the stream of chunks to send to the client.

I was about to suggest presigned urls as well, but for the timespan those are valid they’re indeed shareable. I’m not sure if @benonymus wants that or does really need authentication on each request to the resource.

it depends if they have or not the servers on aws. Outbound traffic between EC2 instances and S3 is free. But sure, from EC2 to client it’s paid (and quite expensive)

@benonymus why you avoid presigned urls?

For streaming using HTTPoison I wrote an article few weeks ago: Download Large Files with HTTPoison Async Requests, but it would be better to use directly a library like ExAws.S3. I can’t find a function to get an Elixir stream from an S3 object though…

Update
Am I wrong or here in the ExAws.S3 there is a massive overhead: https://github.com/ex-aws/ex_aws_s3/blob/master/lib/ex_aws/s3/download.ex#L76 ??

Each chunk seems to be requested with a separate http request…