Fl4m3Ph03n1x
Why is hackney consuming so much resources?
Background
I have an app that makes 2000 requests per second. To make such requests I use HTTPoison which uses hackney. Now each of this requests is fired from a specific pool with 10_000 max connections, so in theory it should be fine.
Problem
The problem is that all 8 CPUs are on fire, and hackney’s mailbox increases steadily overtime - this is translated into a steady RAM usage increase, until there is no more RAM available and bad things start to happen.
Research
To find out what is happening, I used observer and realized that the function with most reductions is hackney_pool:init followed by some others:
Also, the memory graph which indicated the mailbox is rising:
Questions
I am confused. I am not familiar with hackney_pool:init but I assume it initializes a pool (judging from the name) - which is even worst. I only create the pools once at the start, so why is this function being called all the time?
Also, is there a reason on why this is happening? Does it mean that my pool connections need to be bigger?
Marked As Solved
alco
Each row in Observer’s Processes table represents a single process. The second column displays process names for named processes (e.g. code_server, hackney_manager) or the initial func called when the process started if the process has no name (e.g. hackney_pool:init/1, tls_connection:init/1). If you look at the Current Function column, you’ll see that most processes are different OTP behaviours: gen_server, gen_statem, etc.
Back to your question, hackney’s pool implementation has known faults. See https://github.com/benoitc/hackney/issues/510, https://github.com/benoitc/hackney/issues/549. It doesn’t look like hackney’s maintainer has enough interest or time to get if fixed properly, so I would advice on disabling hackney’s pool and setting up your own pool of processes where each process performs a single HTTP request.
Also Liked
Manzanit0
LOL, we’re in 2020. not 2019. I don’t know why I thought this was just this month… Nevermind me 
rvirding
Does the memory usage keep growing? So for example in the next minute would it have reason above 900 Mb? Is the number of processes growing? This you can see on the first System page.
Fl4m3Ph03n1x
Yes, the memory keeps growing. However the number of processes seems to be stable at arround 24% - 26%. What does it mean?
The MsgQ for hackney_pool:init does seem to grow without bound however.
Mmmmm, would you recommend any articles on doing such? I am usually weary of re-inventing the wheel all on on my own :S
josevalim
If you are talking to different ones, then I would suggest getting rid of the pool altogether and just do individual requests that close the connection after the usage.
However, if it is the same website, then a pool is recommended to avoid reopening the same connections over and over again.
The other thing to consider is to actually reduce the number of max_connections. Having a very high limit actually makes things worse in many cases. For example, if the issue is that the upstream website cannot keep up with the load, opening more connections will make everything worse. This does not rule out the other advices in this thread though. It may be all of those factors combined.
Fl4m3Ph03n1x
To the same one. The thing that changes with each request is the parameters.
However I do fail to see how this would matter. Could you elaborate?
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
The deployment platform built for Elixir: PaaS ease, VPS control.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"