hubertlepicki

hubertlepicki

Profiling memory usage in given process

I’ve got some requests misbehaving, and processes handling them suddenly allocate a lot of memory, which can crash my Beam VM.

I can monitor and detect that, I can also enforce max memory usage by process, that’s not an issue.

I would like to, however, know what is eating up the memory. Ideally I’d be able to take a snapshot of memory used by current process, have it dumped to a file that contains Erlang terms, which later I could analyse and find out what’s on stack, what’s on heap etc.

Is there something out there I can use to achieve that?

Most Liked

sasajuric

sasajuric

Author of Elixir In Action

The stack trace should help you narrow down the problematic code. We actually had a similar situation recently, and we ended up periodically logging the processes with unusually high mem usage, together with their stack traces. In this particular situation that info was enough to completely understand the root cause. I agree that it’s not quite what you’re looking for, but it can help you in understanding the problem.

It’s also worth keeping in mind that in a general case excessive mem usage can be caused by other things, such as messages accumulated in the process queue, or refc binaries, or ets tables, so finding the cause is not necessarily as straightforward as logging the current heap of the process, which is why I mentioned a couple of different things in my previous answer.

garazdawi

garazdawi

Erlang Core Team

That is true. However, since you cannot loop without calling functions I’ve very rarely found that to be a limiting factor.

axelson

axelson

Scenic Core Team

You can run observer locally and connect to a remote node to view its information. Here’s a blog post about it: http://jbavari.github.io/blog/2016/03/11/using-erlang-observer-on-a-remote-elixir-server/

sasajuric

sasajuric

Author of Elixir In Action

AFAIK, it’s not possible to get what you’re looking for out of the box. To get some clues, you could use a combination of the following:

  • memory usage of the process
  • process stack trace
  • message queue length
  • total binary memory usage
  • total ETS memory usage

In case you don’t know which process is the offender, you might also need to include :initial_call and :registered_name (both available via Process.info).

Coupled with the knowledge of the code, this should help you narrow down the problem.

hubertlepicki

hubertlepicki

I already solved the issue as in why and what is happening. It was this bug in absinthe we stumbled upon: https://github.com/absinthe-graphql/absinthe/issues/569

I’d like to know if there’s something that’ll help me out next time tracking down such and similar issues :slight_smile:

Where Next?

Popular in Questions Top

albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New

We're in Beta

About us Mission Statement