pdilyard
Extremely high memory usage in GenServers
I’m deep into debugging a very high memory usage problem in a group of GenServers.
There are two types of GenServer implementations I’m examining:
# module:
MessageEngine.Thought
# example state:
%DB.Thought{__meta__: #Ecto.Schema.Metadata<:loaded, "thoughts">, active: false,
score: 0.35795454545454547,
conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
conversation_id: 1621, id: 129158,
inserted_at: #Ecto.DateTime<2017-03-07 21:32:19>,
lost_against: %{"129129" => [51952, 51955, 51955, 51938, 51931, 51951, 51944], ...},
message: #Ecto.Association.NotLoaded<association :message is not loaded>,
message_id: 12748,
text: "Yes because will she listen to them or the people.",
updated_at: #Ecto.DateTime<2017-03-07 21:44:21>,
user: #Ecto.Association.NotLoaded<association :user is not loaded>,
user_id: 51959, vector: [],
won_against: %{"129129" => [51946, 51934, 51934, 51942, 51954, 51957], ...}}
# module:
MessageEngine.User
# example state:
%DB.MessageUser{__meta__: #Ecto.Schema.Metadata<:loaded, "messages_users">,
accepting_choices: false,
all_choices: [%{"c" => 129138, "nc" => 129154}, ...],
comparisons: [%{"a" => 129138, "b" => 129154}, ...],
conversation: #Ecto.Association.NotLoaded<association :conversation is not loaded>,
conversation_id: 1621, id: 132055,
inferred_choices: [%{"c" => 129138, "nc" => 129154}, ...],
manual_choices: [%{"c" => 129138, "nc" => 129130}, ...],
message: #Ecto.Association.NotLoaded<association :message is not loaded>,
message_id: 12748, rid: nil,
user: #Ecto.Association.NotLoaded<association :user is not loaded>,
user_id: 51959}
I don’t want to dig too deeply into why the states are what they are, but suffice to say that they have been well-researched and tested, and I don’t want to explain too much industry context 
Now, we have been monitoring our app in production for a while, and noticed that, as the number of these processes alive increase, memory usage goes up almost exponentially.
With 600 MessageEngine.Users and 600 MessageEngine.Thoughts, we measured almost 35GB of RAM being used across the cluster.
I first tried to measure the amount of memory used just by the state of the process, but this doesn’t seem like nearly enough data to have that substantial of an impact.
I popped into observer to learn more, and ran the following tests:
30 users and 30 thoughts
- With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0
One MessageEngine.User process was consuming 139kb of memory
One MessageEngine.Thought process was consuming 3kb of memory
- With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53
One MessageEngine.User process was consuming 502kb of memory
One MessageEngine.Thought process was consuming 25kb of memory
300 users and 300 thoughts
- With:
length(MessageEngine.User.all_choices) = 0
length(MessageEngine.User.manual_choices) = 0
length(MessageEngine.User.inferred_choices) = 0
length(MessageEngine.User.comparisons) = 0
One MessageEngine.User process was consuming 1089kb of memory
One MessageEngine.Thought process was consuming 6kb of memory
- With:
length(MessageEngine.User.all_choices) = 53
length(MessageEngine.User.manual_choices) = 20
length(MessageEngine.User.inferred_choices) = 33
length(MessageEngine.User.comparisons) = 53
One MessageEngine.User process was consuming 4023kb of memory
One MessageEngine.Thought process was consuming 41kb of memory
So, as you can see, not only is memory usage per-process scaling up a lot just by adding ~50 maps to a list, the usage of each process also seems to be dependent on the number of processes alive! An order of magnitude increase in the number of processes results in an order of magnitude increase in the memory usage of each one.
This seems like really weird behavior to me, and I’m kinda stuck on where to go next, because, by my calculations, the memory usage of the state of these processes should be more like 20-50kb each (used this guide: http://erlang.org/doc/efficiency_guide/advanced.html#id68680).
Here’s a full dump of the state of a process that was using 4023kb of RAM: https://gist.github.com/pdilyard/92a04ccad39be87d05e466ed4dbea193
Any help would be greatly appreciated.
Marked As Solved
sasajuric
It looks like you have one or more processes that are touching a “large” binary (i.e. a binary > 64 bytes), but are not allocating data frequently enough to be garbage collected themselves.
A large binary is reference counted, instead of being copied across processes. A reference count is bumped by every process that touches such binary. When a reference goes out of scope, the count is going to be decremented only after a fullsweep GC takes place. Until then, the ref count of a binary is > 0, and it’s kept in memory even if no one uses it. Therefore, if you have at least one process that touched a binary in the past, but is not allocating data too frequently to trigger a “fullsweep” GC, you’ll end up with an excessive amount of garbage binaries.
A simple example could be a process that acts as a mediator. It receives a message, then dispatches it to another process, and does nothing more than that. It doesn’t allocate a lot of data on its own, so it’s going to be GCed less frequently. If a part of dispatched messages is a large binary, the process touches a lot of large binaries, and can therefore be the cause of excessive dangling garbage.
You first need to identify such processes. Judging by your other output, it looks like they could be your User processes, but I can’t say for sure.
Once you know which processes are causing the problem, a simple fix could be to hibernate the process after every message. This is done by including :hibernate in the result tuple of handle_* callbacks (e.g. {:noreply, next_state, :hibernate}). This will reduce the throughput of the process, but can do wonders for your memory usage.
Another option is to set the fullsweep_after flag of the problematic process to zero or a very small value. I think that GenServer.start_link(callback_module, spawn_opt: [fullsweep_after: desired_value]) should do the job. For more explanation, look for fullsweep_after in docs for the :erlang module.
Also Liked
dominicletz
Bit late but another useful new feature to start your gen_server with the hibernate_after option, such as:
{:ok, worker} = GenServer.start_link(module, args, hibernate_after: 5_000)
This will ensure that once your worker is bored for more than 5 seconds it will garbage collect everything it can.
michalmuskala
I haven’t really closely followed the discussion, so this might be a bit misplaced. But a common pattern for handling memory-expensive operations inside a GenServer is to spawn a separate process to do the processing - this means the process itself does not grow extensively in size, and the memory used for the computation can be freed immediately (when the “operation” process terminates) - you could even consider starting the process with a bigger initial heap to eliminate GC completely (though, that might be risky and excessive without thorough measurement).
For example, this could look like this:
def handle_call(_req, from, state) do
task = Task.async(fn ->
# some computation
end)
{:reply, Task.await(task), state}
end
Or in case the response could be delivered asynchronously, even like this:
def handle_call(_req, from, state) do
Task.start_link(fn ->
# some computation
GenServer.reply(from, reply)
end)
{:noreply, state}
end
dom
bin_leaks forces a garbage collect on all processes, and measures how many reference-counted binaries were freed per process. So this confirms lack of GC is the issue here.
Some things you can do:
- If you have operations that generate lots of refc binary garbage, do them in a separate, short-lived process linked to your long-lived user process, so it doesn’t accumulate garbage.
- You can use a timer to hibernate (see genserver doc) the user process after N seconds of inactivity, or when you know it won’t be getting messages for a while. The process will still be alive, but won’t hold extra memory.
- You can also use a timer to force a gc every N seconds.
- ETS as mentioned can help. Each process can own a table, it doesn’t have to be shared. This is a nice article about the difference it makes: http://theerlangelist.com/article/reducing_maximum_latency
sasajuric
It’s hard for me to give any specific advice, other than not to go for ETS unless you know you need it 
Usual cases for ETS involve multiple processes reading/writing the same data. Another example could be a process with a large active heap which is frequently changing. There are probably other cases, but these are the ones I can think of immediately, where ETS can improve perf/mem usage dramatically.
If you don’t have problems without ETS, then I’d say just stick with that
michalmuskala
The GenServer does little allocations which means it’s heap is kept small - GCs will be more frequent getting rid of the issue of holding on too long to the references to binaries. The binary leak is most prominent with processes that have huge heaps - this can happen if for a normally “quiet” process you have one, infrequent operation that is extremely memory expensive. This operation will cause the heap to balloon, and later will keep the GCs rare in regular operation, since there’s still a lot of free memory left - causing the process to hold on to the binary references for longer than it should.
Keeping the overall heap of the process small will prevent it from holding on too long to those references.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"