I am indeed familiar with Khepri, and I think both it and Ra are important contributions in the direction of strong consistency on the BEAM. I have remarked before that it is quite strange there is no consensus primitive in OTP (e.g. a Paxos implementation), and Ra is literally that. BTW, Erlang is actually older than the first working consensus algorithms (Viewstamped Replication and Paxos).

Ra and Khepri are not, however, sufficient for my goals in particular.

MultiPaxos-style replicated log databases like Khepri are not meant to scale out and are designed to store a pretty small amount of data. Their tradeoffs also require them to store an unnecessary number of copies of the main dataset, which is fine for a small dataset but very bad at scale. Khepri also happens to be an in-memory database (the entire dataset is in RAM), which is not an architectural limitation but a tradeoff they’ve decided to take (which I’m sure is fine for their use-case).

Databases like this (see Zookeeper, etcd, Consul) are generally used as control planes rather than used to store the main dataset. The problem with this approach is that it means you actually have to build an entire distributed database. Something like Zookeeper is maybe 5% of an actual database.

Hobbes inherits from FoundationDB’s architecture. FDB is a reconfiguration system which is explicitly designed to store large datasets but provides a very open-ended data model. So FDB is maybe 80% of a database, but it solves nearly 100% of the “hard problems” of building a distributed database. Correctness is very hard, and FDB provides an abstraction which is correct and scales out of the box.

As I’ve mentioned in the past, I am interested in building tooling to replace things like Postgres, S3, and so on. I need an abstraction which can scale up to “real” datasets so that I don’t have to keep solving the same distributed problems over and over again. I want to solve them once, because they are very hard.

Hobbes is designed to provide strong consistency guarantees while storing several orders of magnitude more data than something like Khepri (and serving equivalently more traffic). Architecturally, the difference in complexity to meet that requirement is quite substantial, but that is what achieves my goals.

If you’re interested in the tradeoffs here, check out this excellent article which covers some of them.

I did not have the knowledge when I started! I acquired it by trying. Of course I had to read a lot and studying databases had become a hobby of mine before it turned into writing databases, but at the end of the day (and this is really true of everything) reading is not enough. There is a lot of stuff that only clicks once you sit down and try to implement it.

Maybe, maybe not. If you’ve followed my posts on this forum you will know that this project was born specifically out of me trying to build apps that are better than CRUD apps with half-working REST APIs and consistently failing to meet my own standards. Some of that is on me, but I prefer to think in terms of process errors and I’ve found that bad tooling leads to bad results.

Especially in this brave new world of AI slop I have become convinced that saas-with-forms is just not going to cut it anymore. I am trying very hard to do better.

Well, the explicit goal of this project is to make sure you don’t have to

But really, if you’re interested in this stuff it might not be as hard as you think! Keep in mind that non-programmers would say the same things about the knowledge you take for granted. Don’t count yourself out.

I will give you the long answer to this question.

First of all, Hobbes is a low-level database; that is, a tool for building databases. The architecture is optimized specifically for a “distributed OLTP” model where the dataset is large but transactions are small. In these workloads, contention is rare and has to be guarded against mainly for correctness. This architecture is pretty optimal for app backends and so on.

However, there are some workloads that have high contention. These workloads do not scale (here’s a fun paper). As it turns out, financial transactions are quite close to the worst possible case here. It is common to route transactions through a small number of accounts which leads to extremely high contention (“hot keys”).

So for financial transactions specifically the best design is to write extremely tightly optimized code to process all transactions in batches as fast as possible on a single core. Elixir is essentially the worst language I can imagine for doing something like this.

Coincidentally, there is a new database designed specifically for financial transactions called Tigerbeetle which was the topic of discussion a few posts up. I mentioned them because I’ve spent a good amount of time in their codebase now (learning) and, frankly, they have the best engineering practices I have ever seen. If you need to store financial transactions Tigerbeetle is without question the way to go. You still need another database to store everything else, though, as they can only store transactions.

Hobbes is a low-level OLTP store and can be used as a tool to build essentially any kind of distributed database. You could use it to build a relational database, or as a metadata store for blob storage (like S3). You could even use it to store metadata for an analytics/columnar data warehouse; this is literally exactly what Snowflake did with FoundationDB (Hobbes’s architectural donor). You could keep track of Parquet files as blobs and query them with something like DuckDB. In fact, that pattern has become a whole thing lately.

But financial transactions are essentially the prototypical example of the worst possible case for a “horizontally scalable” database like Hobbes.

Hobbes is a database, and is definitely not a library for stateful actors. When I say “OTP primitive”, what I mean is that Hobbes resembles a very sophisticated persistent ETS table. Where you might use :ets to back an in-memory data structure, you would use Hobbes to back an on-disk data structure.

There is a bit more, though, because Hobbes is not simply “on-disk ETS”. (Actually we have DETS for that.) Hobbes is fault-tolerant and will scale to very large datasets across nodes. An ETS table is not fault tolerant (tied to one node) and cannot be (natively) sharded. Also, ETS tables do not have transactions at all.

And so this is why I say it’s like a new OTP primitive, because there has never been an OTP primitive which can do these things. The closest would be Mnesia, but it suffers from small shard sizes and poor consistency guarantees.

Maybe Hobbes could be used to build a library for stateful actors. But that’s just because it is a tool for persisting state in general. For example:

• A job library like Oban could use Hobbes to persist jobs (they currently use Postgres)
• An event-sourcing library like Commanded could use Hobbes to persist events (they also seem to use Postgres)
• A distributed process registry could use Hobbes, taking advantage of its fault-tolerance and strong consistency

But the main reason I wrote Hobbes is to serve my own needs. As you know, I want something that can replace Postgres itself (and S3 and similar tools). Hobbes is essentially an abstraction layer which contains all of the hard problems associated with building a distributed database, so that I can reuse it to solve the “easy” problems each time. Much like Erlang solves the hard problems of distributed scheduling and message passing and so on.

I can then use Hobbes to build, say, a distributed filesystem, or a relational database.

Actually, the relational DB already has a name: it will be called Memex. And Memex will probably be the thing most will take interest in, but it does not yet exist.

Hobbes will always be the “pro tool” for those who want to get their hands dirty, but just not “rolling a database from scratch” dirty

Congratulations on the release. I’ve been doing my best to follow the discussions but as this is pretty specialized material, at one point I gave up.

One thing that I would remark is that a few examples could help. F.ex. there was a wave of people on this forum asking for stateful actors, and there were discussions around how useful would that be at all (if an actor persists state that makes it crash on every restart after, that’s actually a worse state of affairs than before) but there are likely multitude of useful usages.

Want to give us an example of a few of those?