the_wildgoose
OTP Semantics / GenServer starting Genserver, how to organise in Supervisor?
Hi, I’m particularly hoping to get the attention of OTP experts like @sasajuric here
I’ve found myself starting to write significant amounts of code where basically I start a genserver, which in turn starts a couple more genservers in the init phase. I then store the pids of these genservers in my genserver and … hope for the best
Its gradually dawning on me that this may not be optimal and I’m having trouble getting my application to stop, which I suspect is due to dangling processes
I guess, 2 questions:
- How should I best rearrange my app to avoid this situation?!
- I presume that it’s sometimes valid to implement as I have done. However, what do I need to do to make it “safe” and idiomatic? Particularly I think I have corner cases at present if the top genserver is shutdown with :normal, possibly also during an immediate/kill shutdown? And I guess potentially other corner cases?
I will try and present a more concrete example.
I was writing a genserver which monitors a serial port. So I made use of the Nerves.UART library, something like as follows:
defmodule UARTMonitor do
use GenServer
def start_link(args \\ []) do
..
GenServer.start_link(__MODULE__, args, opts)
end
def init(args) do
{:ok, pid} = UART.start_link()
:ok = UART.open(pid, args[:uart], some_other_args)
state = %{
uart_pid: pid,
other_state: blah
}
end
...
end
Now, having reviewed some posts by Sasa, and I’m skimming through the Elixir language example and also Elixir in Action ch9 (I confess I read it a long while back and it’s only suddenly starting to make sense now…). These make me think that I should be conforming with a general handwaving guide to always start things through a supervisor, hence my code above feels “wrong”?
Should I have created a DynamicSupervisor and had that call my UART.start_link() ?
However, such a change potentially increases boiler plate quite a bit and leaves me unsure how to always create the semantics that I want?
a) If I want the supervisor to restart my UART process if it dies, I guess then I need a process registry to give me some kind of consistent PID naming (so I can keep calling functions in that process)? However, in the event of spawning multiple processes, it seems sometimes difficult to invent unique dynamic names?
b) I’m not sure how to create the semantics that: if the main process dies, it should kill the UART process also? I guess I would start the UART under (say) a DynamicSupervisor, then just do a “link()” to my own process?
c) What if I needed to guarantee some cleanup that must be done if the UART process stops (not just crashes)? Do I need a third process linked to the UART or can I reliably catch exits from the top process (as it dies) and cleanup the UART process?
d) How does shutdown happen? I can’t get my head around whether it’s enough to start the (say) DynamicSupervisor for the UART after the top level process? Thinking about ensuring hypothetical cleanup runs correctly (imagine it was important to squirt some “bye” message down the UART before closing it)
Going in the other direction, what do I need to do to make the current situation “safe”? Do I need to catch exits? What conditions need to be handled (assuming that if the main process is going down it needs to stop the UART process as well?) Is anything else needed?
There must be some good articles on “how to do this stuff”? It seems like I’m missing some real 101 getting started? It does feel as though things could sometimes be simplified if there was a form of start_link() which would also shutdown dependent processes in the case of a :normal shutdown?
I think I’m on the right track to proceed with (in general) “start everything through a supervisor”. So my next thought is about how to construct libraries and whether there are ways that the library can wrap some or all of these concerns? In general should a library provide some of the supervisor pieces, and if yes, how to offer those in a way that can be inserted into the applications supervisor tree?
I would appreciate some thoughts on how to best structure libraries which need a resource checkout to create some kind of usage handle? So this is your database library and the like. So we need a coordinator which will do some work to acquire a handle, we will create a process to store this handle and do the actual work with the handle. Another process will request the resource.
My thought is that this is:
ResourceAllocator module - does the work to figure out a handle, starts a process through:
ResourceDynamicSupervisor - holds the resource processes
A process needing a handle will call into the ResourceAllocator, which will both start_link the new process into the DynamicSupervisor, but also monitor/link with the calling process (as appropriate).
Is this a good pattern to be using?
If yes, why don’t more libraries include the supervisor part in their implementation? Why didn’t the Nerves.UART library choose to include a supervisor to track these processes? Nothing is black and white, but would it generally be a good strategy to include some kind of dynamicsupervisor in the library?
There is good documentation on Supervisors and Genservers, but I feel we could benefit from more description on good patterns for using these building blocks, especially over how to allocate and cleanup resources at runtime and shutdown. Is it a common pattern to build a new supervisor module which wraps and provides utility functions to start processes (I’m thinking of TaskSupervisor), or is it more normal to keep a separate module for managing this?
If I look at a lot of libraries in the wild it seems like it’s most common to provide just a start_link() function, and leave the library user to figure out how to add to a supervisor tree. However, if I set out to design something which would immediately be stuffed into a DynamicSupervisor, then I would probably structure the implementation quite differently? Possibly even offering a custom supervisor module in the style of TaskSupervisor?
Why is the TaskSupervisor style of implementation not the defacto interface for many libraries? Why are we more commonly offering an interface to start processes and not an implementation to start and supervisor a process in one go? I realise that many cases it will be useful to design a custom supervision strategy, but I sense that a significant number would not?
Anyone else have any good articles on how to build solid apps? How to structure apps which need to start multiple instances of things, how to structure the layers of supervisors and what technique used to manage/wrap the starting of the processes?
Marked As Solved
sasajuric
Irrespective of this topic, I try to avoid a supervision tree in the lib app as much as possible. In most cases it’s better to provide the API for starting a supervisor process and leave it to the user of the lib to inject that process in their own supervision tree. This allows much more flexibility, b/c the client can leverage the supervision tree to start/stop processes when they want. The most frequent exception to this I’ve experienced is a global process registry. If I need that, I’ll typically start it in the lib’s supervision tree, because it keeps the API simpler, with no particular downsides.
Yeah, it’s fine if you have good reasons.
In a well designed tree a parent process should never be brutally killed. However, if you’re not trapping exits, a parent process could receive a shutdown exit signal from its own parent, and exit immediately. The child will terminate a bit after that. However, it’s possible that in the meantime a new parent has already been restarted, and that it has started a new child. So you might end up with a brief period of two incarnation of the child running at the same time. Or, if the child is registered, a new child may fail to start. To prevent this, a parent should trap exits. In this case, the exit signal from its own parent will be converted into a terminate callback (this is automatically done by behaviours such as GenServer).
If a parent is manually shutting down its child in terminate (as it should), then yes, the parent should wait for the child to stop.
IMO a parent process should always be configured with the shutdown: :infinity to prevent this situation (this is btw a default for supervisors). Otherwise, the parent may be killed forcefully, and you end up with the same subtle race condition as explained earlier.
Not that I can think of.
This won’t work as sketched, because you need a sibling proc, not the parent pid. Getting that proc is a bit tricky, because you can’t invoke e.g. Supervisor.which_child from the child’s start_link (it would cause a deadlock), so you need to do it in handle_continue.
This would be more like it. I used to do this at some point, but then I figured that all these extra supervisors don’t bring a lot of benefits, but at the same time the code becomes more complicated.
You don’t need to jump into Parent. You could instead try to implement it yourself by trapping exit, starting the child process directly, and handling exit messages. I basically wrote Parent after becoming fed up with doing this repeatedly in the scenarios where I had to handle multiple children 
Parent API is exposed in layers. Most of the logic is in fact implemented in the foundational imperative Parent module. Mixing in that capability in your own gen_statem (or other behaviours such as GenStage) should be straightforward, both with or without creating a generic Parent.GenStatem.
Also Liked
sasajuric
Usually it’s best if each child is running under some supervisor, because supervisor doesn’t run any custom logic, and so it can’t crash or get stuck. As a result, the fault tolerance of the system will be improved.
That said, there are many situations where this approach doesn’t bring much benefits (if any), while it complicates the code and the process structure. This typically happens when one worker is a logical parent of other worker(s), which seems to be the case in the example you posted. In such cases it’s fine to start workers as direct children of a worker.
When you’re directly parenting other workers, you need to pay attention to a couple of things. First, the parent process should trap exits. The main reason for this is to ensure that
the terminate callback is invoked if the parent of the parent is stopping. In this callback you should stop the child and wait for it to exit. This will ensure that the parent stops only after its children stop, which can prevent some subtle race conditions. For the same reason you should also set the shutdown timeout of the parent process to :infinity. Finally, since you’re trapping exits, you need to handle the :EXIT message in some way, e.g. by restarting a child, or stopping the parent.
I discussed this topic in this talk. I also wrote a library called Parent which can help with such challenges. In particular, if you want to parent children from a worker, Parent.GenServer could be useful. See the caveats section for some tips on writing a resilient custom parent.
the_wildgoose
You are (all) incredibly helpful and I just want to thank you for taking the time to offer your experience here. It’s very much appreciated!
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Supporting innovation across the BEAM ecosystem.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
The deployment platform built for Elixir: PaaS ease, VPS control.
Courses that'll move you from confusion to "Aha, now I get it!"