I’ll turn this question on its head: what sort of features, functionality, and so on are you envisioning you can ONLY get with an Elixir database?

This is the third time this has come up in this thread and I dodged the question the last two times so I may as well answer it properly this time. But I will first reiterate that I am not trying to claim Elixir is somehow the best language in which to implement a database. Rather, I wanted to write a database and I quite like Elixir, so it was the natural choice for me. And this is of course the Elixir Forum, a good place to discuss such things.

But anyway, there are arguments that can be made in Elixir’s favor here. First of all, we should avoid reductive statements about performance. Things are never so simple. When designing a distributed database there are a number of different subsystems to implement, each of which have their own performance characteristics. A storage engine, for example, might benefit greatly from low-level control over exactly how memory is allocated.

But, on the other hand, a lot of the code for a distributed database is business logic (which has very strong correctness requirements but performance is less important) and various forms of message passing. The thing about the BEAM is it’s really, really good at passing messages around. They’ve been optimizing that one use-case for like 40 straight years.

So if you want to implement a database in C++, you have to design an actor model/message passing framework yourself. This is, in fact, what the FDB team did 15+ years ago. And what they ended up with is the purest embodiment of the “poorly specified implementation of half of Erlang” snowclone you can imagine. All the good stuff like preemptive scheduling, process isolation, error handling, all of OTP. It’s all missing. I mean seriously, go and try to actually read the FDB source code. It’s not exactly beautiful.

Put simply, if you were going to implement a distributed database from scratch in C, properly, you would end up re-inventing all of Erlang/OTP/BEAM first. And then you have to catch up to 40 years of their performance improvements. I don’t know about you, but I don’t think I can solo that one lol.

Now there are parts of building a database that are performance-sensitive in such a way that they are slow when written in Elixir. Maybe not as slow as you think; I need to do more benchmarking but I think my concurrency control is within 50% of the performance quoted in the FDB paper, and all I did was naively implement it on top of ETS.

But for those slow parts, we have NIFs. Down the road I can just swap the slow Elixir parts out for C or Zig or whatever if I want to. The BEAM was designed to do this!

Performance aside, though, the advantage for Elixir really comes down to extensibility. This has come up a couple times up-thread WRT “layers”, but FDB was always meant to be less of a database and more of a “tool for building databases”. I think that paradigm could be considerably more powerful in Elixir, as it allows you to write the layers with a level of integration that is hard to achieve with something like FDB. Perf is not enough.

Ideally FoundationDB + SQL compatibility. Sorry for low-effort response but lately I’ve been fangirling over FoundationDB way too much.

What is the issue with SQL, it’s a language and it’s implementation dependent, so you could use it for anything that works on sets which is most data, relational or not.

There are so many SQLish languages out there, datadog, grafana etc that would have benefited from being a subset of SQL with implementation specific operators.

@garrison : After reading more of the thread, I agree that a native Elixir/Erlang/BEAM RDBMS could be a boon to the community (and open source in general), which would bolster the “Persistable Data” point in (Sasa Juric’s Server A vs Server B.

It could be a interesting value proposition (capability) to be able to:

1. Deploy a new app that is all Elixir/Erlang/BEAM without the need for any external dependency (ala postgres).
2. Then be able to just spin up new nodes (cluster) as data/traffic grows – app and data still colocated
3. And if things get really serious, then there is some sort of “Cell Division” process to separate into different clusters (e.g. separate database from web-serving) – whatever data/domain topology makes sense for the project – and do it smoothly.

Yes, I think we all agree that BEAM ecosystem has qualities that no-others have to even be able to support the above.

Side Note: Just mentioning Bryan Hunter + Waterpark as the project recently popped up again in my feeds, and it seems at least tangentially related to this thread.

If DB is relational and I can use Ecto in the app, then I’m not as concerned about the flavor of human-usable query language (or dialect) is used to introspect the DB.

I would still want some sort of story about how devs/ops/dba can safely access the DB like I might do using pgAdmin (read-only) for ad-hoc queries to help debug/solve problems that may arise, or query the data for exploratory reporting-like queries. Whatever that may be. Ecto + LiveView? iex remote connection? Custom language?

So is the idea to take FoundationDB DNA-- ala, logical-key-ordering, key-range partitioned and replicated (LSM Tree powered?) with dynamic range splitting and migrations–, clean/tidy design details for BEAM, and use it as the “direct storage engine” lowest level lego blocks, then to build relational on top of that?

So is the imagination that there would be two deliverables: (1) a distributed and embeddable BEAM-native K/V (not OpenRiak), that has modern improvements to better support the (2) RDBMS for our community?

And that to get adoption, in addition to Ecto integration, Oban support would be required by the large populace of phoenix projects.

The intangibles goals/constraints being:

• Improving the Elixir/Erlang/BEAM ecosystem.
• The go-to data storage for new-apps in our community, not necessarily trying to usurp postgres yet.
• Added benefit for freedom and protection against the DB company “rug pull” inevitability?
• Open the door for new tech to be built on top of this-- Maybe a BEAM Large Object Storage?

Could an initial approach be to (top-down) implement an RDBMS feasibility toy-prototype using FoundationDB as the underlying abstraction to validate the idea, then work to rebuild the abstraction (bottom up) replace FDB?

I say this is all “Cool Beans”, and I hope the seed gets planted.