ashton314

ashton314

What can go wrong with GenServers?

Howdy! I’m a researcher trying to understand better how people use Elixir. I have a fair amount of personal experience with Elixir, but I need more than one data point. :slight_smile: I’m particularly interested in GenServers—particularly how they can go wrong.

Have you ever encountered a GenServer that was sensitive to the order of certain messages? E.g. imagine you had a GenServer acting as a proxy for a web service; you might need to send the server :connect to get a token, and only after that could you send it {:request, body} messages.

When have you had bugs with GenServers?

Most Liked

al2o3cr

al2o3cr

As @derek-zhou points out, message-ordering isn’t generally much of a problem. However, the code you described sounds like it has a different issue that’s more common: using one process when you really want a supervision tree.

Instead of a single “proxy” that tracks all the connections made through it, a “BEAMier” approach would make the initial connect spawn a new process focused solely on that connection.

On the flip-side of “too much state in one process” is “no state in the process”, the dreaded “using GenServers for code organization”. I’ll leave further discussion to the docs.

A gotcha that’s easy to miss until it becomes a headache in production: the restart threshold max_restarts in supervisors needs to be tuned once there are many children. Very frustrating to have 3 crashes in 5s bounce your DynamicSupervisor with 3k children :crying_cat_face:

When that supervisor restarts (or on a cold system boot), you can also run into another scaling problem: the “thundering herd”, where those 3k children start up and ALL try to make a SQL query in the same 10ms. Combine those DB timeouts with a low max_restarts on the supervisor and you are now officially Having A Bad Time.

rvirding

rvirding

Creator of Erlang

As you have already mentioned there are no real guarantees in which order the messages sent from different processes will arrive. If this is really important to you then you will have to synchronise the sending processes in some way.

One thing which is guaranteed is that the GenServer process handles the messages in the order in which they arrive and it only handles one message at a time. So for example when it gets a call messages it will call the handle_call callback and the GenServer toploop won’t handle any messages in anyway until the handle_call returns.

Note however if you in one of callbacks communicates with other processes, this is very very very common, then as there is only one message queue per process (NO there is no way around this) then your messages might become interspersed with more requests coming into the GenServer. It is upto to you make sure your message tagging ensures that you don’t receive any GenServer requests cause then they are gone. Also remember that any messages you leave in the message queue will be picked up by the GenServer toploop and most likely be processed in a handle_info callback.

Remember the BEAM/Erlang/Elixir process semantics is very simple and straight forward, e.g one message queue where everything gets entered in order and there are no interrupts. Its very the KICASS principle. :wink: :laughing:

D4no0

D4no0

One of the dangerous things that can go wrong with genserver is leaking memory when using large binaries. Depending on your orchestration, you can easily take the server down too if you run out of memory.

There are lot of articles on this topic, here is one of them: https://medium.com/coletiv-stories/genservers-memory-issues-ef42cc42e3e9

lud

lud

He’s saying that if your genserver process sends and receives messages, you must ensure that your receive clauses will only match expected messages, and not match messages that are intended to be handled by your handle_call/handle_cast callbacks. Otherwise, as you received the message in your custom code then your handle_call callback will not be called (just because it will not be receive’d by the GenServer module).

derek-zhou

derek-zhou

GenServer handled messages in order; so message ordering is usually not a problem. My usual bugs with GenServer are:

  • start up sequence. For example, GenServer A needs so to be started before GenServer B, but I did not have the right order. Or if the system is not properly designed, who is calling who first may not even be deterministic.
  • dead locks. For examples, GenServer A call GenServer B, and B also calls A back.
  • time outs. in pathological cases, one GenServer could have a large backlog of messages. Calls could timeout.

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
mgjohns61585
Could someone help me? I'm making my first elixir program, number guessing game. I can't figure out how to convert the user's guess from ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

Other popular topics Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement