benbonnet
Trying to understand Genserver memory leak
Getting started with otp; having a sample app which main purpose is to ingest data.
A genserver is subscribed to a given channel and receives load of data passed aroudn through pubsub. Its single purpose is tostore this data in the db (using ecto). It does not store any state.
def init(_args) do
Phoenix.PubSub.subscribe(PubSub, "CHANNEL_NAME")
{:ok, []}
end
def handle_info({:matcher, data}, state) do
upsert(data)
{:noreply, state} # , :hibernate} added, solved memory leak, but ¯\_(⊙︿⊙)_/¯
end
defp upsert(klines) do
Repo.insert_all(
ModuleName,
data, # array of struct
on_conflict: :replace_all,
conflict_target: [:key1, :key2, :key3]
)
end
Its memory invariably increase to some 100Mb, and never lowers. Googling around, found out about {:noreply, state, :hibernate}, and it solved my issue.
But no clue why this memory increase ended up to be permanent. Is it because of Ecto ? About the way the data is passed around (pubsub) ?
Hope the question is somehow relevant; would like to get a better understanding
regards
Most Liked
trisolaran
Sounds like this could be a plausible explanation: Extremely high memory usage in GenServers - #23 by sasajuric
karlosmid
Hi! What helped in my team was to run function as a async Task:
Task.async(__MODULE__, : upsert, [klines])
|> Task.await(:infinity)
In that case, when Task is done, gen server memory related to it is released immediately.
Heads up: this was in combination with when the gen server did repetitive tasks triggered by the timer.
ityonemo
I believe hibernate triggers a GC event.
You can also trigger GC manually using :erlang.garbage_collect/0
ityonemo
It’s not required. but sorry, I don’t have answers for you on when the GenServer gets gc’d. Are maybe your messages coming in more rapidly than you can dispatch them to the database? (Aka do you always have a message in the message queue?). This is just a guess but I think if all of the messages are dispatched, the process goes into a tail call that calls GC, but if there are messages in the queue, the process is only put on ice by the scheduler.
Could also be some weird things with binary references, which are handled differently and have different lifetimes.
Here is specific data how the Erlang GC works, but that is not just for GenServers. Erlang -- Erlang Garbage Collector. If you’re curious you can always check out the source code for the gen module: otp/gen.erl at master · erlang/otp · GitHub + gen_server module otp/gen_server.erl at master · erlang/otp · GitHub
hst337
First of all, unused data will always be garbage collected. The question here is at which point it’ll be garbage collected
Second, here klines get copied into Task process you’ve created. So it might trigger gc in GenServer. But this klines will be in the Task until it dies.
But this is a bad solution, because you don’t need a process here, you don’t need to copy this data. You just need to call the Repo.all and then trigger minor gc
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Producing high quality Elixir screencasts since 2017.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"