sezaru

sezaru

How you handle error logging with "let it crash" philosophy?

Hello everyone,

How do you handle error logging with parts of your system that will trigger errors simply because you are using the erlang “let it crash” philosophy?

For example, for my system, I use HoneyBadger to log errors so I can identify them more easily and be notified when a newer error occurs.

The issue for me is that my system connects to multiple third-party WebSockets, and the connection can be lost because the other side closes the connection after some hours.

When that happens I receive a new error entry at HoneyBadger, but at the same time is not an error that I really need to “fix”, since the WebSocket process will simply crash and recreate the connection.

So, how would you handle this? Will you try to handle this in the code so that if the connection is lost, it will not generate an error log (maybe generate a warn log instead) so it doesn’t pollute my log with errors that don’t need to be fixable?

If that’s the case, wouldn’t that be the opposite of the “let it crash” philosophy since I would, in the end, handle that crash?

Marked As Solved

stefanchrobot

stefanchrobot

I treat such events as non-errors. They shouldn’t pollute logs with errors or push stuff to the error reporting tool, otherwise there’s a tendency to start ignoring issues that are real errors.

I usually try to handle them gracefully which gets easier the more the “error” occurs. Alternatively, you can ignore the error. In Ruby, most error reporting SDKs had an option to ignore certain exceptions since error handling is exception-based. The benefit of such approach is that the app does not even attempt to push the errors to the error service. Seems like in Elixir you mostly need to resort to silencing them in the error service which needlessly uses the quota.

Whichever way you pick to “silence” the issue, it’s useful to consider putting a metric on top of the non-error event in question because a big increase in the amount of silenced errors is probably a symptom of a real issue. Funnily enough, pushing the errors to the error service and ignoring them there might be the easiest way to have an actual metric on how often the issue occurs :crazy_face:

The way I think about the “let it crash” philosophy is that it frees you from convoluted error recovery scenarios. But that works under the assumption that the operation will be retried either automatically (a retryable background job) or by the user (a web request) or that you don’t care. So I would consider these scenarios to follow the general idea of “let it crash”:

  • Calling Repo.get! in a retryable Oban job to get an entry from the DB which is expected to exist,
  • Pattern matching on a specific return value of a function in a one-off script, e.g. {:ok, item} = foo(params),
  • Calling Repo.rollback in a Phoenix action midway transaction when something is off and some changes were already applied.

Also Liked

al2o3cr

al2o3cr

There’s a difference between “handling” a crash by trying to keep the crashing process alive and fixing the cause of the crash, and “handling” it by noting that it happened and possibly restarting the process.

Where Next?

Popular in Questions Top

pmjoe
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
itssasanka
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
grych
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
639 49522 488
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

We're in Beta

About us Mission Statement