There are two things to take into consideration as well:

① Restart strategies, specifically :max_restarts - the maximum number of restarts allowed in a time frame. Defaults to 3 and :max_seconds - the time frame in which :max_restarts applies. Defaults to 5. If something went south and there were 4 attempts to restart it within 5 seconds interval (defaults,) all failing, the GenServer would end up not restarted. This is likely your case, if e. g. the DB connection has dropped for 3 seconds. The culprit would be 500ms interval in init/1, which makes 4 consequtive insuccessful attempts to execute do_some_work/1 to fit into 5 secs and shutdown the process forever.

② The mailbox overflow. This technically might be your case, but extremely unlikely, just saying for the complete picture.

The app will terminate if the root pid exits, and immediatelly on the first exit, no restarts. And if the app is :permanent it’ll make the whole vm stop.

Yes the supervisor will exit if it has to restart a child too many times. And it will be restarted by its own supervisor.

You can have a chain of Sup → Sup → Sup → Sup → Worker and all elements of the chain will behave like this.

A good way to think about this is that the code describing the supervisor tree (each supervisor children, and the children description in each child supervisor, and so on) describes “how the app state should be at runtime”. And the system will try its best to maintain that state, or otherwise fail and shutdown totally.

Of course that description can be altered at runtime by calling start_child, delete_child to add/remove parts of the tree, and of course having :transient, :temporary or :significant children.

But there is no way it will skip some parts of the tree unless you tell it to. Max-restarts and al’ will only have a temporary impact on the state of the system, but the app will always get back to the desired state.

So if my understanding is correct, if MyWorker was linked to another process, and that process exited with a :shutdown (or {:shutdown, value}), then it would indeed cause MyWorker to terminate normally as well. Unfortunately, MyWorker isn’t linked to any other process.

If the child spec for MyWorker has restart: :transient then yes! (:shutdown is not “normally” but the supervisor will not restart it indeed).

Any progress on finding the cause?

• Another possibility is that the child spec is overriden somewhere and :transient is replaced with :temporary ; but I guess this is unlikely.
• Are there any calls to Supervisor.terminate_child or GenServer.stop in your codebase?
• Are you using :significant with :any_significant or :all_significant?

Hi Welcome back!

Is there exit(:normal) or exit({:shutdown, some_value}) somewhere in your code that could have been called by that process?

Or maybe it was linked to another process that exits with shutdown? I don’t remember how GenServer behaves in that case.

Thanks for the reply, I think you nailed the two most possible cases that I haven’t though of!

No, there’re no explicit exits that can be called from this process (unless I missed something). I also doubt that Ecto and HTTPoison have such exits in their codebases that could leak into user’s code.

As for how GenServer behaves in this case, I think this part of Process.link/1 docs is relevant:

When two processes are linked, each one receives exit signals from the other (see also exit/2). Let’s assume pid1 and pid2 are linked. If pid2 exits with a reason other than :normal (which is also the exit reason used when a process finishes its job) and pid1 is not trapping exits (see flag/2), then pid1 will exit with the same reason as pid2 and in turn emit an exit signal to all its other linked processes. The behavior when pid1 is trapping exits is described in exit/2.

So if my understanding is correct, if MyWorker was linked to another process, and that process exited with a :shutdown (or {:shutdown, value}), then it would indeed cause MyWorker to terminate normally as well. Unfortunately, MyWorker isn’t linked to any other process.