teqr
Logging stops completely due to Logger crash
This question might dive into sub-areas, so I would like to clarify as much as possible.
Application:
The application has multiple genserver process started though Supervisor and Dynamic supervisor. Application also uses elixir Logger, LoggerFileBackend to log into different files.
Behaviour:
Sometimes some of the process crash also (and recover successfully). Sometimes Logger process crashes completely and logging in all the files stops. This is completely random, happening at stretch of months, or sometimes multiple times in a day. There is no pattern or logs which might indicate the reason and it is not recoverable (for the logger process). However, the application runs fine (for months without logging) and there are no logs or any crash report during this event. This issue only gets resolved after restarting the application (which is not acceptable).
Observations:
As there are no log reports, so this is based on debugging which I was able to do:
- From application iex shell, I can confirm that
Process.whereis(Logger)returns nil. - I assume that some genserver might be crashing multiple times repeatedly - causing Logger to crash entirely. The application genserver is able to recover, however, the logger doesnt recover. This is only my assumption as I dont get any crash report for any genservers of application during this event. Other genserver crashes do get logged properly (when Logger has not crashed).
Environment:
FROM erlang:25.1-alpine
ENV ELIXIR_VERSION="v1.13.4" LANG=C.UTF-8
extra_applications: [:lager, :logger, :runtime_tools]
My questions:
- Is there any better way to configure logs, so as to identify the reason for Logger crashes?
- Assuming due to some reason, Logger crashes inside my application. Is there any way to restart it (without restarting the complete application)?
Most Liked
dimitarvp
Flying blind would worry me. As a first measure I’d do my best to find why, even if it takes a while. As a second measure I’d fire up a separate process that periodically checks if the logger is alive and try to start it if it’s dead; I reckon once every 5 seconds is enough.
But really, don’t you get any stack traces? If you configure f.ex. Loki then all the stdout and stderr outputs get put in the logging backend of Grafana and you’ll see everything.
LostKobrakai
Enabling sasl logging might also help depending on which version of elixir/otp you’re running
hauleth
That is not surprising, as there is no such process at all in Logger. It seems that the Logger in :registered_names in logger.app file is just an omission. You can easily test that by running:
$ iex
iex> Process.whereis(Logger)
nil
My bet there is that either there is some message that handler was removed because of some failure, or there is message that there is too many messages in the process inbox and you run in the drop mode.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Catch errors, track performance, monitor hosts and more.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"