Welcome to the forum @spurgus!

If the response includes the content-length header then you can check that from a response step:

req =
  Req.new()
  |> Req.Request.prepend_response_steps(validate_content_length: fn {req, resp} ->
    with [header] <- Req.Response.get_header(resp, "content-length"),
         {content_length, ""} <- Integer.parse(header) do
      if content_length > @max_content_length do
        Req.cancel_async_response(resp)
        {req, RuntimeError.exception(message: "content-length too large")}
      else
        {req, resp}
      end
    else
      _ ->
        Req.cancel_async_response(resp)
        {req, RuntimeError.exception(message: "Invalid content-length")}
    end
  end)

Otherwise, you’ll have to keep track of received bytes and halt the request. You mentioned streaming but didn’t say which form of streaming (into: :self, into: &fun/2, into: collectable). Here’s an example for the function form of streaming:

Req.get(req, into: fn {:data, data}, {req, resp} ->
   resp = Req.Response.update_private(resp, :length, 0, & &1 + byte_size(data))

   if Req.Response.get_private(resp, :length) > @max_content_length do
     {:halt, {req, RuntimeError.exception(message: "content length too large")}}
   else
     {:cont, {req, resp}}
   end
 end)

Try:

{:cont, {req, update_in(resp.body, &(&1 <> data))}}

I believe this into: &fun/2 option is envisioned for scenarios where you’ll be doing something with the data as it’s coming in (such as sending it to another process), rather than accumulating it and processing it at the end.

I was not meaning to imply this use of into: &fun/2 is wrong, just pointing out Req doesn’t accumulate the body for you with this option (maybe it should?).

Yes, that’s my concern, someone trying to take your server down, making you download huge files, so my idea was to use streaming to abort the download as soon as the max size has been detected.

Let’s see if I can get this to work…

Never trust headers unless you’ve validated them.

Once I had an Elixir bot running in a hostile environment (Kodi addon ecosystem) . I am quite sure it would not have survived the first week if I trusted the headers. Content-lengths spoofing (read: simply returning an everlasting stream of random bits) was one of the first concerns.