What I’ve been doing in the past, in a larger system I’ve worked on, was to introduce another supervision layer, let’s call it the parent supervisor, above the supervisor which supervised the problematic process, let’s call it the child supervisor.

The parent supervisor starts the child supervisor with the restart option set to temporary. This way the child supervisor still does it’s job as usual, until sh*t hits the metaphorical fan and it goes down. As it was started as temporary the parent supervisor won’t restart it, and acts as a circuit breaker.

Of course you either need some monitoring on top - so a human operator can step in and fix the problem - or some kind of exponential restart strategy. But at least you’ve now isolated the issue to part of your system.

Hope this helps?

Your suggestion is precisely what is recommended by the Elixir in Action book:

Opting for the :temporary strategy also means that the parent supervisor won’t be restarted due to too many failures in its children. Even if there are frequent crashes in one child process, say due to corrupt state, you’ll never take down the entire tree, which should improve the availability of the entire system.

You’ll need to handle at least the part of “now stop trying” manually. Supervisors are meant to keep trying – they operate under the assumption that human intervention might need a technician driving to the device.

That’s why you always want something OS level as well like heart or systemd.

I know there is a really good talk from Fred Hebert about how to design robust systems

I think it is this:

https://ferd.ca/the-hitchhiker-s-guide-to-the-unexpected.html

but not sure as I think the graphic was different in the one I am thinking of.

Other useful blogs:

https://ferd.ca/it-s-about-the-guarantees.html