Fl4m3Ph03n1x
Tracing all calls to handle_info in all processes using recon_trace
Background
Our app is constantly printing an error message and we have no idea where it is coming from. My objective is to detect which process(es) is receiving the message so I can then trace it using Process.info via a remote shell and find out which module is causing the issue.
Objective
To achieve this, I am following a suggestion given in a previous discussion and I am trying to use recon.
The error message has the following format:
=ERROR REPORT==== 17-Jun-2019::09:29:22.694288 ===
Unexpected message: {#Ref<0.3271557848.1968439297.87877>,badarg}
After reading the documentation of recon_trace I concluded I needed to base my code on the following example, which traces all calls to handle_call for module Mod for all new processes, and those of an existing one registered with gproc:
recon_trace:calls({Mod,handle_call,3}, {10,100}, [{pid, [{via, gproc, Name}, new]}
In order to try the library I decided to go with a simpler approach and just trace 10 calls from anywhere to handle_info, but it errors out:
:recon_trace.calls({:_, handle_info, 2}, 10)
** (CompileError) iex:3: undefined function handle_info/0
(stdlib) lists.erl:1354: :lists.mapfoldl/3
(stdlib) lists.erl:1355: :lists.mapfoldl/3
(stdlib) lists.erl:1354: :lists.mapfoldl/3
Question
I am connection to the application’s terminal via a remote shell and I am trying to run these commands there.
What am I doing wrong?
Marked As Solved
Fl4m3Ph03n1x
Thank you everyone for your kind feedback. I have taken @NobbZ code and fixed it into the following (the original version was missing a map):
traces =
:code.all_loaded()
|> Stream.map(fn({module, _path}) -> module end)
|> Stream.filter(&:erlang.function_exported(&1, :handle_info, 2))
|> Stream.map(&{&1, :handle_info, 2})
|> Enum.map(&:recon_trace.calls(&1, 10))
This allows me to trace the 10 calls to all modules using handle_info.
Now, we can see the following output:
11:28:07.155854 <0.285.0> memsup:handle_info(time_to_collect, {state,{unix,linux},
true,
{2950590464,31605788672},
{<0.6127.0>,10784136},
false,60000,30000,0.8,0.05,<0.286.0>,undefined,undefined,[],[]})
11:28:07.156521 <0.285.0> memsup:handle_info({collected_sys,{2968354816,31605788672}}, {state,{unix,linux},
true,
{2950590464,31605788672},
{<0.6127.0>,10784136},
false,60000,30000,0.8,0.05,<0.286.0>,
#Ref<0.960887702.3327131654.82864>,undefined,
[reg],
[]})
11:29:07.190019 <0.285.0> memsup:handle_info(time_to_collect, {state,{unix,linux},
true,
{2968354816,31605788672},
{<0.6127.0>,10787544},
false,60000,30000,0.8,0.05,<0.286.0>,undefined,undefined,[],[]})
11:29:07.190603 <0.285.0> memsup:handle_info({collected_sys,{2987864064,31605788672}}, {state,{unix,linux},
true,
{2968354816,31605788672},
{<0.6127.0>,10787544},
false,60000,30000,0.8,0.05,<0.286.0>,
#Ref<0.960887702.3327131656.11201>,undefined,
[reg],
[]})
Which confirms this solution is working !
Also Liked
NobbZ
According to the documentation, that has nothing to do with preloading all modules:
[if the application] is started as permanent, […] the node will shut down if the application crashes permanently.
NobbZ
You probably want :handle_info. lower case identifiers are atoms in erlang syntax.
NobbZ
From recons documentation it seems to me, that the module isn’t allowed to be the wildcard matcher:
https://ferd.github.io/recon/recon_trace.html#calls-3
Moduleis any atom representing a module
NobbZ
You could use :code.all_loaded/0 to get all loaded modules and iterate over them to check if they export handle_info/2 and set a trace programatically…
traces = :code.all_loaded()
|> Stream.filter(&:erlang.function_exported(&1, :handle_info, 2))
|> Enum.map(&{&1, :handle_info, 2})
:recon_trace.calls(traces, 10)
Of course this only works if you make sure to load all modules, eg. via starting a release.







