harmon25

harmon25

Can a GenServer state be too "big" and general application architecture

Hoping the old adage that no question is a dumb question holds true with this one…

When building an application that requires maintaining state, should that state be maintained by a single GenServer process? Or should it be split up across multiple processes?
At what point does the state maintained within a single GenServer become too big? Possible?
Does this just boil down to semantics and code organization?

I guess poor performance would be an indicator one might want to split up the state across more processes. Are their more defined best practices in this regard?

Do mnesia or ETS have a role to play in applications which have a lot of state to maintain, but do not require persistence?

Hope the answers to this question will not only help me better understand and build Elixir applications, but other beginners as well!

Thanks all

Most Liked

sasajuric

sasajuric

Author of Elixir In Action

I wouldn’t say there is one true way. It really varies from case to case, depending on what you want to achieve.

The benefit of keeping the state in a single process is that you have strong consistency. At any point in time, there can be only one process accessing the state. On the flip side, since all the requests are serialized, that process may become a sequential bottleneck if used frequently by many different processes. Another important downside is that you lose the entire state when the process fails. The consequences of a single error might be larger than you want.

Splitting the state/responsibilities over multiple processes give you a reversal of the properties above. There is no strong consistency: you can’t read a frozen snapshot of states from multiple processes, nor can you atomically change states in them. On the upside, you have possible performance improvements, and better failure isolation: if one process crashes, the others are still running, so you get to keep most of your service.

Hence, I’d say that choosing one or the other depends on the problem at hand. The work that needs to be done atomically and consistently should reside in a single process. It’s better off splitting independent tasks into multiple processes, for better error isolation as well as performance and scalability. It’s of course never as simple or as black/white as this, but these are the general guidelines I’d recommend. Occasionally diverging from that path for good reasons is fine.

One point about this:

Does this just boil down to semantics and code organization?

Processes are not the tool for code organization, but rather the organization of runtime. If you need to organize the code, modules are the way to do it. Let’s say for example that the state of your process becomes quite complex, but you still want to keep it in the same process. Then, consider implementing the state in one or more separate, (usually) purely data-oriented modules, and have GenServer functions just use those modules.

JEG2

JEG2

Author of Designing Elixir Systems with OTP

Because I had to look that up, I though I would shared that it means, “There ain’t no such thing as a free lunch.”

NobbZ

NobbZ

We have a saying in germany, that there are no dumb questions, but only dumb answers :wink:

I think this is very similar to a discussion we recently had in a programming exercise (imperative) about a global super “object”¹ to hold and maintain state or having multiple small distinct global variables. There hasn’t been a real conclusion, but after half an hour everyone decided to use multiple small states, to reduce levels of idirection.

I don’t know about best practices here, I do not have that much battle proof experience with BEAM, but depending on how your super state is structured I do fear, that it will massively influence garbage collection (and generation ;))

Haven’t used mnesia so far, but using :ets in my erlang applications all the time. Reads and writes are much faster than a GenServer keeping a Map as state. Also there are optimisations in :ets for concurrent reads and writes (even interleaved), which you can’t do with a corresponding GenServer and a Map.

¹: Super-sized-struct with accessors would be a better wording here, since we were using C.

rvirding

rvirding

Creator of Erlang

ETS is very good for managing a large amount of state when you don’t require persistence. The thing to remember is that it is a datastore and not a database so it does not support transactions, or very,very,very limited transactions. Very limited. So if you need to control access to an ETS table then you need to wrap it with a process.

Depending on what you need a GenServer might the right way to go. Seeing you are keeping the data in an ETS table the Genserver process itself will not get very large. This is basically what Mnesia does except that it provides a very large set of features, for example transactions, replication, distribution, persistence. It gives you a distributed ACID database.

TANSTAAFL which means you have to decide what you need and what you are willing to pay for.

One thing to remember with ETS tables is that as they are stored “outside” all processes accessing them means copying data between the process and the table.

Robert

gregvaughn

gregvaughn

You had to look it up? Never read Heinlein? Hand in your nerd card. :smiley:

Where Next?

Popular in Questions Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
sergio_101
I am VERY much an elixir newbie. I have taken one elixir course and one phoenix course on Udemy. During that course, I saw the instructor...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New

We're in Beta

About us Mission Statement