Announcing EctoFoundationDB 0.1.0!

EctoFoundationDB is an Ecto adapter for FoundationDB, a distributed key-value store that is designed to be scalable, fault-tolerant, and performant.

Quick Links:

• Documentation
• Livebook Guide
• GitHub
• Hex.pm
• Why FoundationDB

Features:

• CRUD plus indexes
• Multi-tenancy
• Automatic migrations
• Custom indexes
• FDB Transactions

Due to FoundationDB’s Layer Concept, EctoFoundationDB is a more than a wrapper. It has opinionated default behavior that is intended to fit the needs of modern web applications, and it also allows you to add structure to your data beyond the table. It does both of these things with ACID transactions, to ensure your entire data model is in a consistent state no matter what.

For example, maybe you want to put all your Users in a durable queue to process later. Or maybe you’re interested in implementing your own vector similarity search directly on top of your existing data model. Perhaps you’re intrigued by the sound of automatic schema migrations. Maybe you just need some very solid simple data storage with high availability.

EctoFoundationDB can help you do any of this.

For me, after managing various medium-to-large-scale SQL and NoSQL databases in production for 12 years and eventually deciding I’m more of a NoSQL guy, I simply wanted an Ecto adapter where I felt like I was at home.

Finally, thanks to @Schultzer and @warmwaffles for their open source adapters. I learned a lot from ecto_qlc and ecto_sqlite3, and you should definitely check them out!

Hi there, thanks for the message. You’ve rightly identified that there are some risks to running erlfdb or ecto_foundationdb for a project. Let’s discuss from the bottom-up.

FoundationDB server and libfdb_c: Maintained and released by the FoundationDB team at Apple. Production ready, battle tested. There are reasons to choose FDB over other DBs and reasons not to. Happy to discuss more, but probably out of scope for this post.

erlfdb: A NIF wrapper of libfdb_c. With any NIF there is risk of bringing the BEAM VM down. The project was originally implemented by the CouchDB team working closely with the FDB team. The apache-couchdb/erlfdb project is used in production apps with success.

The foundationdb-beam/erlfdb fork (where the hex.pm package comes from) has some changes, and to my knowledge has not yet had a production deployment anywhere. However, I’m aware of one project where it will be soon, in an app that’s very important to me professionally.

I’ve been conservative with my changes to the fork to preserve its production-readiness. I’m confident erlfdb will hold up well to production scrutiny. Of course please report any bugs to the issues page.

ecto_foundationdb: Still young, and ready for experimentation. No battle testing to my knowledge, but I do seek to change that. There are some projects that I have in mind, but they’re still a ways out.

In FDB parlance, ecto_foundationdb is a Layer. A consequence of an FDB stack is that correctness in the Layer is just as important as correctness in the database itself, so extensive testing is encouraged. An example of something that needs more testing focus is migrations. Everything works on paper, but it needs a longer term app to live in to make sure the migrations hold up as expected across iterative application releases.

In short, I’d call erlfdb production-ready, but not yet production-proven (due to the fork) and ecto_foundationdb is ready for community experimentation. I am personally and professionally invested in them both, and welcome further discussion, issues, and PRs.

I actually believe the FDB design decisions work quite well for small and moderately sized databases as well. For me, it’s more about the kind of workload and the operations story. I wouldn’t want people to come away thinking that they need 100TB of data in order to use it – LiveSecret is essentially a 0-Byte database running on a couple GB of memory (please don’t try to DOS it though ). I’m pretending FDB is an embedded database by using ex_fdbmonitor , instead of SQLite, even though I am a big fan. Of course, please check FDB’s system requirements for a production app. Running with low system memory does go against the grain of a recommended FDB deployment. (8GB per fdbserver process recommended)

To date, the largest value add for me is never having to write DDL and never having to worry about migrations while I’m doing additive changes (new schemas, new fields, new indexes), which is 99% of development anyway IME.

We wouldn’t know until we tried it, to be honest. Some Ecto features come for free, and some require the adapter to meet specific contractual details. But if it doesn’t work, I imagine it would be possible to add support. Let me know if you find out!

Happy to report there is progress on the main branch, not yet in a release. Specifically, there is a new CLI module, a guide, and a test for deleting or renaming a field safely, in the spirit of Safe Ecto Migrations. Renaming a field in this way actually requires 2 full table rewrites, which you may find unfortunate, but at least now it’s possible to do it .

Other progress on the main branch includes refactoring the IndexInventory into a general purpose metadata store. This paves the way for richer metadata such as a schema field map (like an atom table) that could allow for table operations without rewrites, similar to what’s been described in this thread. That may come in future work – I’m still considering the implications.

v0.5.0

Notable Enhancements

• Index Metadata [doc] now makes use of FoundationDB’s \xff/metadataVersion key,
  which allows the client to cache metadata and maintain transactional isolation
  without having to wait on any keys. Thanks @garrison for pointing this out!
• Versionstamp [doc]: Added the ability to insert objects with a monotonically increasing integer id, via FoundationDB’s versionstamp.
• SchemaMetadata [doc]: This is a new built-in Indexer that allows your app to watch and sync a collection of objects in a tenant. (Demo: Sync Engine Part II - Collections)

Changelog

Nice! I was looking at possibly building a FoundationDB adapter, but I haven’t used it before to really know what I was getting myself into.