larshei

larshei

Best way to persist a processes state?

Hey everyone!

Preface

Over the last days I started to create an online version of Doppelkopf, a german classical card game, using Elixir/LiveView. The project is still in a pretty basic shape and deployed on fly.io - feel free to check it out at fehlsolo.de

Players join rooms and then create games. Both rooms and games are just GenServers.

When a GenServer dies, it dumps its state to an Agent and tries to fetch that state when it restarts. Works quite well so far.

When a game ends, the final state is dumped to a file as a binary (:erlang.term_to_binary). The score screen then grabs the file and calculates scores etc - in future this allows to watch a replay of the game, take over control at any point, analyse the games and maybe even train some models with it … if I ever get that far :smiley:

Challenge

When e.g. a new version of the app is deployed, all process state is lost and all rooms and games disappear. As the game results are also just stored in files in the container, they are lost as well.

Discussion

What would be your preferred way of persisting state between app restarts and for the “long run” (game history)?

Possible approaches?

What I thought about is:

  1. Store the process state as files, but in a cloud storage
    Not sure about the (dis)advantages of this approach.
  2. Store the files to a fly volume
    The fly docs themselves suggest this is not a good idea for anything that should be around “longer” - might be fine to persist process state during an update.
  3. Store the process state in postgres as bson
    What is the best way to turn the retrieved (nested) map from the database back into the structs of my application? I have Ash available, but so far, all the “models” are just structs, not Ash Ressources or Ecto.Schemas.
  4. Create normalized models and store everything in the db
    I really dont want to do this :smiley:

How would you tackle this?

Marked As Solved

cevado

cevado

You don´t need to have it normalized… you can use whatever you are already using to identify the rooms/games, make a json field(jsonb type if postgres, jsonb blob if sqlite) that stores the whole nested state as it is.
you can cast whatever comes from the db to a struct with struct/2 or use a schemaless changeset, or an embed schema if you have any sort of nesting to help cast the nested stuff together.

Also Liked

al2o3cr

al2o3cr

Consider what I’d call “option 3B”: store the state in a database table that has a BLOB column with an :erlang.term_to_binary in it.

This avoids the extra complexity of serializing/deserializing into other shapes.

It does make the contents of the state very opaque to DB queries, since doing binary-to-term in SQL would be quite challenging.

cevado

cevado

you need to keep in mind that it gonna be a migration that needs to, retrieve data from the db, transform it in elixir, write back to the db. it’s preferable for your migration to do all the work as just sql statements, sometimes you can’t avoid the back and forth but in this particular case you could avoid it by just making it a unstructured column that the db knowrs how to handle.

so, when you do %MyStruct{field_a: 1, field_b: 2} you’re actually calling a function MyStruct.__struct__(field_a: 1, field_b: 2), this function is defined when you do defstruct .... This function is the thing that ensures that the output map has all the fields of the struct and only those fields and no other stuff.
so if the struct definition changes between when you do term_to_ binary/1 and binary_to_term/1 you gonna end up with a map without the guarantees of the __struct__/1 function call. you can test this on iex:

defmodule A, do: defstruct [:a, :b]
bin = :erlang.term_to_binary(%A{})
defmodule A, do: defstruct [:a, :c]
:erlang.binary_to_term(bin)

the risk of doing that is that all function that you pattern match on %MyStruct{} gonna pass, but the struct can end up not having the shape that you expect.

if there is a need to use binary_to_term/1 with structs, I strongly suggest using Map.from_struct/1 and struct/2 functions to ensure at least a few of the guarantees of what you expect from a struct are kept.

cevado

cevado

it’s off-topic but I see migrations as db “book keeper”, any relevant change or important operation I like to keep it as migration for historical purposes. in case is something to be executed manually, i’d just add the timestamp of the migration to the schema_migrations table and that’s it. but i don’t think this applies here

seeplusplus

seeplusplus

Adding my thoughts on this to help OP and future readers out. I have done the :erlang.term_to_binary cast into db column approach for storing game state before and it works lovely. You can even :zlib.zip/:zlib.unzip too, if you like. To the three things you’ve pointed out:

  1. For my case at least, this isn’t an issue. YMMV, but I think it’s likely that, like OP, many people will be in an “I just need to get this working” state, so they won’t care about this. If you do ever need it, you can extract the fields you need as separate columns*. Then you can use a migration, written in Elixir, to migrate existing data. You even have options here too - you can do a “bulk” update where you change all the records at first app boot up or you can do a “live update” where you do this migration when records are accessed/written.

* Another possibility is reporting on the metric you’re interested in elsewhere. For example, when making a clone of the game Boggle, I stored boards and user’s submitted words as erlang binaries in blob columns. I wanted to do reporting on the number of “globally new” words found by users every time they finished a game. Rather than changing the structure of existing, working tables, I just did the reporting to Prometheus. This is a highly specific to me example, but I wanted to share other ways I think are valid to work around this issue.

  1. It will certainly not be as easy as UPDATE SET field WHERE clause, but it can be done. How one does this depends on a lot (size of records/tables/concurrent users). When building a small game, chances are there are few users and hence large chunks of time you can afford to have an inconsistent DB. This is highly dependent on all of the factors mentioned above, because if your records are small, you have few users and few games, such a migration script and its associated inconsistency might last a few seconds on the extremely long side. Even if this isn’t the case, there are ways around that - they aren’t easy, but it is possible.

  2. I’d like to hear more about this if you can give an example. I’m fairly new to Elixir and my understanding is that structs are basically just maps with syntactic sugar.

seeplusplus

seeplusplus

I think you’re probably right in the general case, but I think in OP’s case - a card game with likely single to double digit users and low data volume - this is unlikely to be an issue and certainly I wouldn’t say “there is no way” to do this. There are just some risks and if folks understand those risks, then they can find ways to work around them in the interim until they feel like moving to an approach that more closely resembles what you’re describing.

Where Next?

Popular in Questions Top

jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
Fl4m3Ph03n1x
Background Let’s assume I have a typical GenServer that receives messages as requests, does some operation in a DB and returns responses....
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
vonH
In asking this question I am more interested about the expressiveness of the language itself and less concerned about the availability of...
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
gshaw
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
New
malloryerik
Hi, this is for people who, like me, have had some friction using .html.heex templates in VSCode. The solution seems to be, in a hyphena...
New
aadeshere1
I have a another noob question about loop. Since elixir is immutable, while loop is not directly possible. total = 10 while total != 0 ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
beno
I will often find my self writing things similar to: case some_value do nil -> something() "" -> something() _ -> someth...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New

We're in Beta

About us Mission Statement