Hi, author of the quoted article here.

1. To my knowledge, there are no such libraries. I wouldn’t necessarily use one anyway because what needs to be restarted, when, and under which conditions, is not necessarily super easy to make generic. I’ve written some that would “diff” supervision trees and be used to “repair” configuration calls that were missed, and I’ve written some that could just do a cooldown. Some would restart workers, some would restart supervision trees wholesale. I’ve had some that had no automation, but relied on an operator sending a command to restart a thing (it acted as a fuse for major cluster meltdowns). Making this kind of stuff generic kind of implies very flexible monitoring and linking schemes with arbitrary logic, and at that point a GenServer or gen_statem are plenty to go from.
2. I just call one of the supervisor module’s functions (either to restart a child, or to delete the old one and add a new one). I do this from another OTP process.
3. The general circuit breaker will be to detect and react to faults, react to timeouts, etc. I would use a circuit breaker a lot when I expect failures from the other component rather frequently, especially when there is a need for coordination of fault detection between all workers (i.e. all DB workers may want to expect the remote DB being down and to avoid thundering herds). The “config manager” in our case was to cope with supervision trees where each worker connects to a distinct resource, but each worker could also be started, created, dropped, by remote users interacting with our gateway. Since we already had a need to “repair” the config (say the network is down during 1-2 weeks and we don’t see config changes), so it was simple to bolt the retry feature on top of it. The difference is that really, we wanted to be able to add some smarts to our supervision strategies, whereas a circuit breaker is more of a general overload/flow control mechanism.

I said these things in the other thread but I guess its worth repeating again.

Your service should be able to boot without any of its dependencies. That means that any dbs, queues, or external services can be 100% unavailable and your app should still start. As I said in the other thread if you can’t do this then your app will be brittle and less reliable. The root of your problem isn’t your supervision strategy. The root problem is that you’re assuming these external services will be mostly available. You need to be more pessimistic.

There isn’t a single supervision strategy, library, circuit breaker, or pattern that is going to make this work for you. Those are all tools. But in order to know how to use those tools you have to start from first principles and design your system to work even when the rest of the world is burning to the ground. That’s why having the ability to start your application without having access to any of its dependencies is a good heuristic for a system that can withstand failure. It means that you aren’t truly dependent on those systems and if you have a transient failure you’ll be in better shape to recover from it. At the very least it points your design in a more reliable direction.

Meta note: this probably didn’t warrant a whole new thread and could have been continued in the original one so as not to lose context.

1. I’m not aware of any, but I’d suggest also looking for erlang ones, which I expect to more likely yield results.
2. It can be as simple as a GenServer or gen_statem, depending on how you want it to function. It’s most likely using monitors to keep knowledge about if a certain important process is running and to be notified if it crashes. Another option would be linking itself to important processes, but trapping exits. I’d personally use monitors, because that’s what they’re for. How exactly you deal with the knowledge of if things are running or down or crashing is up to you. Also how the process knows what things should be up and running is up to the implementation.
3. I’d not say that circuit breakers are a replacement for the “configuration manager”. They handle totally different tasks and maybe you even want to use both.
   A circuit breaker does monitor a call into a subsystem and if the number of failing responses exceeds a certain threshold it blocks calls into the subsystem by short-circuiting into an error. Depending on the library there are then certain ways to heal from a blown circuit, which can be time based or with some backoff, maybe only a fraction of calls is let trough and enough successful ones make it go back to normal. A circuit-breaker does nothing for the subsystem’s healing besides blocking request.
   The “configuration manager” process on the other hand does not block anything. Its sole purpose is to monitor processes it knows about and maybe restart them based on some logic in the implementation. It basically handles the “healing” part of a subsystem.

And some addendum:

Your usecase is a supervisor with lot’s of workers holding connections afaik. If you let your supervisor die it’ll take down all those connections, even the healthy ones. So you don’t want to let your supervisor be temporary, but rather your workers. Or add another layer between both.

It does, but this only works for direct calls to those crashing processes. Often requests to your system also involve e.g. pre-processing before calling into the volatile subsystem. With a proper circuit breaker you can stop requests before doing any pre-processing in the event the subsystem is not working. It basically allows you to short circuit in any layer on top of the actual failing subsystem instead of just at the edge of calling into said subsystem.

Edit: As your “configuration manager” does have knowledge about if a certain process is running or not, it could also act as a curcuit breaker if it has an API, which let’s other processes query for the status it knows about, but I’d not like to put a process, which is meant to be as stable as possible in such a potentially hot path as a circuit breaker switch.