A small update on this.

My original (experimental) intent was to have a consistent distributed in-memory database, which would also automatically be able to automatically manage dynamic clusters (addition / removal of nodes) and to recover from net splits.

I have made strongly eventually consistent systems that automatically configure themselves before (like syn), however consistency is of course a whole different beast.

ram v0.2.0’s operations used global locks and transactions with a 2-phase commit. However, in case of net-splits I’ve basically reached a point where I can ensure strong consistency… on the split sub-clusters. Which basically means that the system as a whole is only eventually consistent.

This is kind of a simple thing really: if a system is set to auto-configure itself, it doesn’t really know which ones of its node are there to stay and those that are not (i.e. differentiate net-splits from node decommissioning), so it cannot define what consistency means. I’ve tried playing with some algorithms but I wasn’t really going anywhere useful.

So, I’ve followed @RudManusachi and @fabriziosestito’s suggestion and used Ra in the current ram implementation. This means of course that cluster definition (along with node additions and removals) need to be explicitly set with sys-op tooling. So far, I’m happy of the results.

If you wish, you can check the new ram docs:
https://hexdocs.pm/ram

As usual, feedback and discussion are welcome. Thanks to everyone that took the time to be part of this thread.

Yes @kokolegorille and @ostinelli , CubDB is an embedded database, therefore by design not distributed. In principle one could use it as a backend for a distributed system, but its main use case is embedded applications (think Nerves, or a mobile or desktop application), or for cases where one needs persistent and fail safe application-local storage.

It goes beyond key-value: it supports sorted selection of ranges, and atomic transactions. It is stored on disk, and optimized for robustness in case of sudden shutdown and for needing very little memory and CPU (all good qualities when running on small embedded devices).

You could evaluate it as an optional durable backend for your distributed K/V store, although in your case DETS is probably simpler to adapt if you only need K/V

So, I’ve tried using GitHub - rabbitmq/ra: A Raft implementation for Erlang and Elixir that strives to be efficient and make it easier to use multiple Raft clusters in a single system.

I’ve found some minor quirks (like inconsistencies when using ra:start() vs application:ensure_all_started(ra), or ra:overview() blowing up if ra is not started), but overall the main issue I can see with my main requirement of simplicity of use is that ra needs specific handing of cluster initialization & changes. Just like mnesia, in some way.

You can’t simply add / remove a node automatically, you need to specifically manage its addition to an existing cluster, and you can only do so one node at a time. You can’t simply “merge” nodes of different clusters for instance, afaik. This requires some level of sys-op tooling. It’s not an issue per se’ of course, it’s just that I’m looking for something simple to use and deploy.

Also, I’ve tried reading about how to manage conflict resolution handling in case of net-splits, but I couldn’t find a specific place where to look at. Admittedly I haven’t spent a lot of time searching for this one in particular, so I may have just missed it.

My little experiment ram handles all of these things btw, but it probably has other issues that are instead perfectly handled by ra.

This part of the README covers dynamic clusters.
Only thing to keep in mind is that only one cluster membership change at time is allowed, concurrent changes will be rejected by design (this is described in the Raft paper too if I am not mistaken). This means you have to wait for the clusters to propagate the cluster membership change prior changing it again.

About consistency guarantees, Raft guarantees strong consistency.

Have a lot of fun!

There is one in Elixir