Fl4m3Ph03n1x

Fl4m3Ph03n1x

Tracing all calls to handle_info in all processes using recon_trace

Background

Our app is constantly printing an error message and we have no idea where it is coming from. My objective is to detect which process(es) is receiving the message so I can then trace it using Process.info via a remote shell and find out which module is causing the issue.

Objective

To achieve this, I am following a suggestion given in a previous discussion and I am trying to use recon.

The error message has the following format:

=ERROR REPORT==== 17-Jun-2019::09:29:22.694288 ===
Unexpected message: {#Ref<0.3271557848.1968439297.87877>,badarg}

After reading the documentation of recon_trace I concluded I needed to base my code on the following example, which traces all calls to handle_call for module Mod for all new processes, and those of an existing one registered with gproc:

recon_trace:calls({Mod,handle_call,3}, {10,100}, [{pid, [{via, gproc, Name}, new]}

In order to try the library I decided to go with a simpler approach and just trace 10 calls from anywhere to handle_info, but it errors out:

:recon_trace.calls({:_, handle_info, 2}, 10)
** (CompileError) iex:3: undefined function handle_info/0
    (stdlib) lists.erl:1354: :lists.mapfoldl/3
    (stdlib) lists.erl:1355: :lists.mapfoldl/3
    (stdlib) lists.erl:1354: :lists.mapfoldl/3

Question

I am connection to the application’s terminal via a remote shell and I am trying to run these commands there.
What am I doing wrong?

Marked As Solved

Fl4m3Ph03n1x

Fl4m3Ph03n1x

Thank you everyone for your kind feedback. I have taken @NobbZ code and fixed it into the following (the original version was missing a map):


traces = 
  :code.all_loaded() 
  |> Stream.map(fn({module, _path}) -> module end) 
  |> Stream.filter(&:erlang.function_exported(&1, :handle_info, 2)) 
  |> Stream.map(&{&1, :handle_info, 2}) 
  |> Enum.map(&:recon_trace.calls(&1, 10))

This allows me to trace the 10 calls to all modules using handle_info.

Now, we can see the following output:

                                         
11:28:07.155854 <0.285.0> memsup:handle_info(time_to_collect, {state,{unix,linux},
       true,
       {2950590464,31605788672},
       {<0.6127.0>,10784136},
       false,60000,30000,0.8,0.05,<0.286.0>,undefined,undefined,[],[]})
                                          
11:28:07.156521 <0.285.0> memsup:handle_info({collected_sys,{2968354816,31605788672}}, {state,{unix,linux},
       true,
       {2950590464,31605788672},
       {<0.6127.0>,10784136},
       false,60000,30000,0.8,0.05,<0.286.0>,
       #Ref<0.960887702.3327131654.82864>,undefined,
       [reg],
       []})
                                          
11:29:07.190019 <0.285.0> memsup:handle_info(time_to_collect, {state,{unix,linux},
       true,
       {2968354816,31605788672},
       {<0.6127.0>,10787544},
       false,60000,30000,0.8,0.05,<0.286.0>,undefined,undefined,[],[]})
                                          
11:29:07.190603 <0.285.0> memsup:handle_info({collected_sys,{2987864064,31605788672}}, {state,{unix,linux},
       true,
       {2968354816,31605788672},
       {<0.6127.0>,10787544},
       false,60000,30000,0.8,0.05,<0.286.0>,
       #Ref<0.960887702.3327131656.11201>,undefined,
       [reg],
       []})

Which confirms this solution is working !

Also Liked

NobbZ

NobbZ

According to the documentation, that has nothing to do with preloading all modules:

[if the application] is started as permanent, […] the node will shut down if the application crashes permanently.

https://hexdocs.pm/mix/Mix.Tasks.App.Start.html

NobbZ

NobbZ

You probably want :handle_info. lower case identifiers are atoms in erlang syntax.

NobbZ

NobbZ

From recons documentation it seems to me, that the module isn’t allowed to be the wildcard matcher:

https://ferd.github.io/recon/recon_trace.html#calls-3

  • Module is any atom representing a module
NobbZ

NobbZ

You could use :code.all_loaded/0 to get all loaded modules and iterate over them to check if they export handle_info/2 and set a trace programatically…

traces = :code.all_loaded()
|> Stream.filter(&:erlang.function_exported(&1, :handle_info, 2))
|> Enum.map(&{&1, :handle_info, 2})

:recon_trace.calls(traces, 10)

Of course this only works if you make sure to load all modules, eg. via starting a release.

kip

kip

ex_cldr Core Team

Deleted so the future historians will wonder what amazing insight should have been here

(but @NobbZ will know it was because made an totally incorrect statement)

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
jc00ke
Expanding on this topic: https://forum.elixirforum.net/t/map-typespec-question/19217 Let’s say I have a map with required and optional k...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

Other popular topics Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New

We're in Beta

About us Mission Statement