Hello, I’m struggling to work through memory usage problems when trying to run many queries at once using Task.async + Task.await. I’m relatively new to Elixir + Phoenix, so bear with me.

I need to fire off multiple POST requests to a third party service at once and collect all of the responses. Here’s a simplified version of what I’m doing now.

[
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  Task.async(fn -> some_func_that_makes_a_post_req(...) end),
  ...
]
|> Enum.map(&Task.await/1)
|> Enum.reduce(...) # do something with the data returned

I’m using hackney to send the POST requests

:hackney.post(
   "some_url",
   [
     {"Content-Type", "application/json"}
   ],
   Jason.encode!(query_body),
   [:with_body]

This seemed to work well, but once I started profiling my application I noticed massive memory spikes. My Phoenix application uses around 60-80 MB of memory at baseline, but when I spawn ~40 Tasks (which each make 1 post request) I see a massive spike up to 150-200 MB, and the processes seem to hang around for a minute or two.

Using the :observer I was able to track down which processes were hogging memory. I can’t upload images to the elixir forum yet, but I’m seeing numerous ssl_gen_statem:init/1 processes consume ~2 MB each. They stick around long after the post requests have completed and each task has been awaited.

Any ideas on what I’m doing wrong? I want to be able to send many requests at once, parse each response & format them independently, and finally merge all of the results into one Map / struct. Is there a better way to send numerous https requests at once without flying too close to the sun & running into OOM issues?