Well, yes, overload protection and ensuring the handlers survive is one thing, the other thing is how the system behaves and if it survives when the output of logs exceedes the capacity to process them.

Logger has several options to handle that situation, the ones you may want to tweak are :sync_threshold, :discard_threshold and :max_buffer.

Basically we had situation where logger was switching to sync mode, and stayed in it too long, the I/O was stuck and we just decided never to do it because the whole system would freeze, and switched to discard mode instead of ever hitting sync mode was the best option for us.

The elixir library doesn’t implement overload protection, however if you use erlang logger backend, the backend will have overload protection?

Does that make sense? I am not entirely familiar with logger backends, but it seems that should be the principle.

Yeah, the choice of wording is strange indeed, maybe there is something more to it.

It might also be that erlang did not have overloading protection back in the day when this was written down in elixir documentation, but this is just a theory.

I guess you have to wait for a elixir core member to answer.

If I am reading the documentation right, the Logger will switch to sync mode after 20 messages in the buffer, then will wait for 480 more messages before it switches to discard mode. On the nodes that are not doing a lot of logging that means these requests are stuck for that time.

https://hexdocs.pm/logger/1.12.3/Logger.html

The docs you referred to are for 1.12.3, but we’ll be going to 1.16.1, so here:
https://hexdocs.pm/logger/1.16.1/Logger.html

The section in question is the one about Erlang/OTP handlers:

https://hexdocs.pm/logger/1.16.1/Logger.html#module-erlang-otp-handlers

which states that:

At the moment, there is no built-in overload protection for Erlang handlers, so it is your responsibility to implement it.