harmon25

harmon25

Can a GenServer state be too "big" and general application architecture

Hoping the old adage that no question is a dumb question holds true with this one…

When building an application that requires maintaining state, should that state be maintained by a single GenServer process? Or should it be split up across multiple processes?
At what point does the state maintained within a single GenServer become too big? Possible?
Does this just boil down to semantics and code organization?

I guess poor performance would be an indicator one might want to split up the state across more processes. Are their more defined best practices in this regard?

Do mnesia or ETS have a role to play in applications which have a lot of state to maintain, but do not require persistence?

Hope the answers to this question will not only help me better understand and build Elixir applications, but other beginners as well!

Thanks all

Most Liked

sasajuric

sasajuric

Author of Elixir In Action

I wouldn’t say there is one true way. It really varies from case to case, depending on what you want to achieve.

The benefit of keeping the state in a single process is that you have strong consistency. At any point in time, there can be only one process accessing the state. On the flip side, since all the requests are serialized, that process may become a sequential bottleneck if used frequently by many different processes. Another important downside is that you lose the entire state when the process fails. The consequences of a single error might be larger than you want.

Splitting the state/responsibilities over multiple processes give you a reversal of the properties above. There is no strong consistency: you can’t read a frozen snapshot of states from multiple processes, nor can you atomically change states in them. On the upside, you have possible performance improvements, and better failure isolation: if one process crashes, the others are still running, so you get to keep most of your service.

Hence, I’d say that choosing one or the other depends on the problem at hand. The work that needs to be done atomically and consistently should reside in a single process. It’s better off splitting independent tasks into multiple processes, for better error isolation as well as performance and scalability. It’s of course never as simple or as black/white as this, but these are the general guidelines I’d recommend. Occasionally diverging from that path for good reasons is fine.

One point about this:

Does this just boil down to semantics and code organization?

Processes are not the tool for code organization, but rather the organization of runtime. If you need to organize the code, modules are the way to do it. Let’s say for example that the state of your process becomes quite complex, but you still want to keep it in the same process. Then, consider implementing the state in one or more separate, (usually) purely data-oriented modules, and have GenServer functions just use those modules.

JEG2

JEG2

Author of Designing Elixir Systems with OTP

Because I had to look that up, I though I would shared that it means, “There ain’t no such thing as a free lunch.”

NobbZ

NobbZ

We have a saying in germany, that there are no dumb questions, but only dumb answers :wink:

I think this is very similar to a discussion we recently had in a programming exercise (imperative) about a global super “object”¹ to hold and maintain state or having multiple small distinct global variables. There hasn’t been a real conclusion, but after half an hour everyone decided to use multiple small states, to reduce levels of idirection.

I don’t know about best practices here, I do not have that much battle proof experience with BEAM, but depending on how your super state is structured I do fear, that it will massively influence garbage collection (and generation ;))

Haven’t used mnesia so far, but using :ets in my erlang applications all the time. Reads and writes are much faster than a GenServer keeping a Map as state. Also there are optimisations in :ets for concurrent reads and writes (even interleaved), which you can’t do with a corresponding GenServer and a Map.

¹: Super-sized-struct with accessors would be a better wording here, since we were using C.

rvirding

rvirding

Creator of Erlang

ETS is very good for managing a large amount of state when you don’t require persistence. The thing to remember is that it is a datastore and not a database so it does not support transactions, or very,very,very limited transactions. Very limited. So if you need to control access to an ETS table then you need to wrap it with a process.

Depending on what you need a GenServer might the right way to go. Seeing you are keeping the data in an ETS table the Genserver process itself will not get very large. This is basically what Mnesia does except that it provides a very large set of features, for example transactions, replication, distribution, persistence. It gives you a distributed ACID database.

TANSTAAFL which means you have to decide what you need and what you are willing to pay for.

One thing to remember with ETS tables is that as they are stored “outside” all processes accessing them means copying data between the process and the table.

Robert

gregvaughn

gregvaughn

You had to look it up? Never read Heinlein? Hand in your nerd card. :smiley:

Where Next?

Popular in Questions Top

New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement