I’m willing to bet there’s something wrong with your application code (or instrumentation! I’ve made that mistake before). I work on a service that will make hundreds of downstream requests and we definitely don’t see any big CPU bumps, so something seems off

Also I don’t know what kind of resource allocation your application has available in terms of CPU or memory, is it seriously constrained? Still, I wouldn’t expect a 20%-50% bump. Could it be the way you’re manipulating or merging the data? Though that seems pretty unlikely given the payload size.

In short, something fishy is going on! You might need to try doing some profiling. I’m guessing you can reproduce it locally? The Observer is a great tool as well if you’re not already using it.

There are a few things that you might think about, however they don’t explain the CPU bump:

• Make sure you’re passing SSL certs in as a file, here’s an example in HTTPoison but similar options exist in Finch/Mint/etc. If you’re making enough downstream requests it can affect memory usage.
• We switched from HTTPoison to Finch and found a big improvement in terms of reliability and speed, looks like you’ve tried it already, just wanted to endorse it
• Your code example makes me think you could simplify it with Task.async_stream (although I would also consider Task.Supervisor.async_stream or Task.Supervisor.async_stream_no_link, depending on your needs). I’m pretty sure async_stream matches your use case exactly, but maybe I scanned your post too fast!

I don’t know if connection pools can help in this regard, but give it a try.

Since all the calls to external APIs are HTTPs, it would be great to hold onto those connections for subsequent calls. (The decryption and encryption might be maxing out the CPU usage)

See Finch:

https://hexdocs.pm/finch/Finch.html#module-usage

Here’s how Plausible Analytics uses it:

Example:

defp finch_pool_config() do
    finch_pool_config = Application.fetch_env!(:derpy_coder, DerpyCoder.Finch)
    default_pool_config = finch_pool_config[:default_pool_config]

    %{
      :default => [
        size: default_pool_config.size,
        count: default_pool_config.count
      ],
      "https://derpycoder.site" => [
        protocol: :http2,
        count: 50,
        conn_opts: [
          transport_opts: [
            timeout: 15_000,
            verify: :verify_peer,
            cacertfile: mkcert_path("/rootCA.pem"),
            keyfile: mkcert_path("/rootCA-key.pem")
          ]
        ]
      ],
      "https://s3.derpycoder.site" => [
        protocol: :http2,
        count: 50,
        conn_opts: [
          transport_opts: [
            timeout: 15_000,
            verify: :verify_peer,
            cacertfile: mkcert_path("/rootCA.pem"),
            keyfile: mkcert_path("/rootCA-key.pem")
          ]
        ]
      ]
   }
end

Isn’t it a good thing to max out the CPU though?

You are enqueuing all the requests in one instant in your loop, then it’s expected that the system will run as much as possible concurrently. If I had enough memory I would like the CPU spike to be at 100%, to handle as much work as it can the faster it can.

If you want to smooth things down you can use Task.async_stream and tweak the :max_concurrency option to limit concurrent jobs, at the cost of higher response time.

Finch uses connection pools, so you could use that to limit the concurrent connections as well, but I am not sure how that translates to control concurrent requests. That should be available in Req as well, as it uses Finch under the hood.

Are the connections using HTTPS? That’s got a significant initial overhead to negotiate the initial SSL connection, regardless of how short the request + response are.

You are basically awaiting each task one by one instead of doing Task.await_many or, as @lud said and it’s even better: use Task.async_stream and have its max_concurrency be less than or equal to the maximum size of your Finch pool.

Not sure if that’s going to bring down the CPU load by a lot but it will help somewhat + you will be doing even better parallelization.