hubertlepicki

Memory leak (binaries) that only :recon.bin_leak/1 helps with

I have an app that is misbehaving. It’s deployed to Gigalixir and there are two problems with it as far as I can tell: one is that it has mysterious memory spikes that are causing OOM / killed problems, the other one is a slow binaries leak when I make HTTP requests to download external files.

While I am struggling with the first issue, maybe the solution to the second will soft out the first too. So the bin leak issue can be easily observed on the app. Freshly after it’s deployed, Phoenix LiveDashboard shows minimal memory usage:

Now, the only thing I have to do to cause trouble is to download some files, and it doesn’t matter how I really do it. I can use HTTPoison with sync or async response, or I can use Mojito / Mint. I don’t have to write files to disk, I can ignore them. I can download them in a spawned process / Task, and never care about return value. The memory used by BEAM on the server just keeps growing.

For example, when I do several HTTP requests like this, to download a 17MB file, and completely ignore it:

iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok 
iex> task = Task.async(fn -> Mojito.get(url, [auth]); :ok end); Task.await(task)
:ok

I can see the BEAM memory growing in both AppSignal and LiveDashboard:

and then after several more requests:

The system has a low number of fullsweep_after_flag set, and I tried setting it to 0 with no difference:

iex> :erlang.system_flag(:fullsweep_after, 0)
0

So the memory keeps growing in the “Binaries” section, and I am pretty sure it’s the HTTP response (that I ignore) accumulating and eating up the memory.

Interestingly, manuallly triggering GC run does not help either. :erlang.garbage_collect() has no effect.

The only thing that does help to release the memory is running :recon.bin_leak/1

So I do have a binaries leak, and I suspect it’s nothing specific to my code but rather a configuration / setting / flag issue.

The binaries stay at around 250-300 MB and never get cleaned up unless I manually kick in :recon.bin_leak/1.

Since this app is running on a low memory instance (0.6GB), this is a real issue as it experiences occasional crashes because it runs out of memory.

I am using Elixir releases, with Elixir 1.1.2 and Erlang 23.1. As part of this adventure I already upgraded from Elixir 1.1.1 and Erlang 22.2, where the issue was the same.

Any ideas? I am running out of hair to pull.

20 comments

#beam #memory-leak

28 3087 20

2021-01-19 14:27:36 UTC

Marked As Solved

axelson

Scenic Core Team

Do you get the same issue locally in development? I wonder if there’s a bug in mojito. Do you get the same memory leak with finch (https://github.com/keathley/finch)?

Post #4

Also Liked

ferd

Author of Property-Based Testing with PropEr, LYSE, & Erlang in Anger

it’s the way to investigate and solve the core issue though.

The bin_leak call should tell you which process is shedding the more load because the only thing it does is:

a) look for the binary memory for all processes in the node
b) run a GC in all processes on the node
c) look for the binary memory for all processes in the node (again)
c) output the top ones

by looking into which processes are shedding load you can easily find which operations are problematic. It seems from context here that it would be a pooling issues for data going through it, but as documented in Erlang in Anger, there are also ways to work around that (a common one being to hibernate to compact memory usage, and another one being to avoid some copies of binary by sending other information around, which finch likely does here).

I would advise in general to dig and understand the pattern you see ongoing, because it will help you gain the knowledge and experience to prevent it from happening within your own code in the future, and also to know at a glance whether a library you’re about to add to your project is likely to show the same problems before you actually try it in production.

Post #15

sneako

Thanks! I’m very proud to see this growing interest in Finch. I definitely agree that streaming request bodies is a useful feature to add. I haven’t had too much free time to look very closely at you PR yet (my daughter was born on Tuesday ), but the usage of Tasks is sticking out to me as something I think we should try to avoid, since this could cause the same kind of inter-process copying that we do not want. Are you sure that they are necessary?

Post #10

hubertlepicki

I don’t see binaries growing when using Finch

Post #6

sneako

Really cool to see that Finch solved this problem! My best guess as to why it did is that Mojito uses poolboy, where the pool worker always owns the connection. Request data has to be sent and copied into the worker process and then the response has to be sent and copied back to the caller. With Finch, and NimblePool, we just transfer connection ownership to the caller, which allows you to skip the copying between processes.

Post #7

hubertlepicki

Yes, good job on Finch.

Interestingly, both Mojito and HTTPoison had this issue showing up for me, and both of them use Poolboy and copy over the binaries as you say.

I am in the process of switching the project to use Finch at the moment and so far it looks very good. One issue I have is that I need to send streaming multipart requests with large files, currently doing that streaming with Hackney, but I am working on Finch support for streaming requests (opened WIP PR here https://github.com/keathley/finch/pull/105), and I am using https://github.com/teamon/tesla/blob/master/lib/tesla/multipart.ex to build the actual Stream containing the request body, multipart-encoded that can be streamed with Finch just like I do with Hackney. So far so good.

In long run, I’d love this code being pulled out of Tesla to separate library, so my dependencies are lighter.

So my idea is:

add support to streaming requests to Finch (WIP PR is there)
extract Tesla.Multipart to stand-alone lib
write some documentation on how to marry the two to perform streaming multipart requests the way I am doing them

Post #8