Hey, someone else has been doing this?! Looks like beautiful work! I’ll have to spend some time looking through the code later.

So glad to see FDB’s architectural influence growing. The design is just so good, and it maps really well onto the BEAM; probably because they FDB guys were somewhat inspired by Erlang/Actor model.

Bedrock 0.2.0

Major Features:

• KeySelector API: Added comprehensive key selector support across storage, gateway, and repo layers with range query capabilities
• New Olivine Storage Engine: New B+ tree(ish) storage implementation with advanced indexing and persistence
• Range Reads: Full range-read support with key selectors, conflict detection and read-your-writes within transactions
• Enhanced Transaction Processing: Refactored commit proxy finalization with synchronous sequencer notification and exactly-once reply guarantees

Architecture Improvements:

• Reworked transaction tree balancing and processing
• Improved resolver capabilities with enhanced telemetry and tracing
• Director recovery system with capability-based retry mechanisms
• Shale log consistency fixes and transaction stream enhancements

Developer Experience:

• Updated documentation with transaction format guides and architecture deep-dives
• Subspaces and fast tuple packing/unpacking
• Enhanced example bank livebook with range query demonstrations
• Comprehensive test coverage for new features

This release represents a significant evolution of Bedrock’s transactional capabilities with improved performance, reliability, and developer ergonomics.

Hey folks

Quick update on Bedrock! Just released v0.3.0-rc4 with some major API improvements based on community feedback.

What changed:

The transaction API got much cleaner. Instead of threading a transaction handle everywhere:

# Before - explicit transaction threading
transaction(fn tx ->
  put(tx, "user:123", user_data)
  get(tx, "config:app")
end)

# After - implicit context + Ecto-style returns
{:ok, config} = transact(fn ->
  put("user:123", user_data)
  config = get("config:app")
  {:ok, config}
end)

Subspaces are no more! The new Keyspace system replaces the old Subspace concept and features built-in encoding support:

# Works great with the directory layer
{:ok, app_dir} = Directory.create_or_open(["app"])
users = Keyspace.partition(app_dir, "users", key_encoding: Tuple, value_encoding: BERT)

# Familiar things are still here
alice_key = users |> Keyspace.pack({"alice", 42}) # still makes keys, but you get more control over *how*

# gets / puts / etc. use the Keyspace as context for understanding how to encode and decode your data
%{name: "John", age: 23} = Repo.get(users, 42)

:ok = Repo.put(users, 42, %{name: "John", age: 23})

# Range operations will decode and values for you, too, if you want:
Repo.get_range(users) |> Enum.to_list()
[
  {42, %{name: "John", age: 23}}
  {63, %{name: "Jane", age: 25}}
]

Also simplified conflict management, improved error handling, and streamlined the range operations.

Why this matters:

These changes make Bedrock much more ergonomic for daily use while keeping all the other goodness intact. The API now feels more “Elixir-native” rather than being a direct port of FoundationDB patterns.

The encoding system is particularly nice - you can plug in custom serialization for keys/values while the keyspace handles encoding automatically. Great for when you want structured data but it gets out of your way if want to manage binary keys and values directly. It’s made the class scheduling example very succinct.

This version focuses on the concrete improvements users will see, shows before/after code examples, and explains why the changes matter for real applications. The suggestions here have been phenomenal, and I’ve tried my best to integrate them all.

Always interested in more feedback, especially if you’ve worked with FoundationDB or other distributed stores!

This looks very cool – congrats on the milestone release!

Thanks! I’m an admirer of your ecto-foundationdb project… It kind of pushed me into building this, actually.

I ran through your Livebook and unit tests, and read some code, and everything worked as promised. Excited to see where this project goes – do you have an recommended starting points for an interested potential contributor like myself?

I’m glad to hear it all worked!

Absolutely there are places to contribute. I’m working on a list of things in the “issues” on github, and I’m very open to more ideas. I have a bunch of write-ups in the “guides” section of the repo that breaks out how the various components interact and how processes (like recovery) are working currently. I’m actively working to improve the docs to make the project more accessible.

Things that could be fun:

• An ephemeral version of the log / storage servers (simple, backed by :ets) that could be used by end-users to run unit tests without needing to configure disk space, etc.
• Storage recovery. There’s no mechanism to replace/clone a completely failed storage instance.
• Data Distribution. There’s no mechanism to split or coalesce shards, and and :dets (which underpins the example storage engine I have in there now) is limited to ~2gb, making the system as a whole limited to ~2gb.
• Testing. It isn’t glamorous, but this is a db, and people are going to expect it to work. There’s a fair bit to do here. I plan to spend a bunch of time working on this to make sure that what’s already done works.
• Configuration. I can foresee some mix tasks to allow easy access to change the parameters and configuration of a running instance. It’d be pretty dope if you could start out with an ephemeral single-node system and configure it into a multi-node cluster without stopping it.

What are your thoughts on a storage engine? Is it possible to give the same Elixir treatment to FDB’s redwood?

The storage system is very pluggable. I’ve done some research into Redwood, and I think we can actually do better in some ways. The state of the art has advanced considerably in the db world since it was introduced. I’d like to try mixing some of the ideas behind Aerospike (in-memory indexes) and TiDB (Blob-backed storage with LSM tree compaction outside the critical path).

Another thought was that it might make sense to have multiple types of storage / log engines at play, at once. You could imagine a memory-only engine for high-speed reads. Another that persists data out on blob-stores like GCS for extreme durability. Another (like redwood) that would allow versioned reads going back further than 5s, to support snapshot reads that run much longer while remaining consistent.

Last thought, as the Layer guy: The cool thing about Data Layers is that it should be totally reasonable to swap out FDB with an API-compatible backend like this one, with a little glue code.

This was also my thinking. I could very much see adding higher-level Layers like sql (perhaps based on ecto_foundationdb), job queues like Apple’s QuiCK, etc. It would be wonderful if we could have these things all work within the same transactions – a very exciting prospect.

Thanks for taking an interest!

It certainly could be done that way, but there are some drawbacks. The biggest is API ergonomics - Take an operation like get(key). Unlike a Map, the get operations here have side-effects. (The fastest storage servers are located and cached, etc.) If done in a purely functional style, the return from get would need to look like {result, tx}. Pipelining would be extremely clunky for operations like put, breaking the idea that Bedrock is like some really big, durable, transactional map.

There’s an argument for using the process dictionary instead of a GenServer, for similar effect, but spinning up a new process is pretty cheap with the BEAM. I have some ideas that require managing timers and timeouts that pretty much require the use of another process… so, we’ll see what the future holds.

Here’s how it works. The first read gets a read version from the sequencer. All of the subsequent reads for this transaction (and all nested transactions) will use this same version. This one of the ways that Bedrock can fulfill it’s guarantee of repeatable reads and snapshot isolation: All reads for a key at a version will return the same value.

When a transaction is committed, if any reads were performed, the read version is used along with the keys that were read to check to see if any transactions modified those keys in the intervening time – this is part of what the Resolver does. If the keys that were read have been modified, the transaction is aborted and restarted (and it will then pull a higher read_version and so will see the newly modified data).

The sequencer is a cluster wide singleton, but it’s job is extremely simple. FoundationDB, the pattern on which Bedrock is based, uses the notion of “GRV” proxies, read-version proxies that will spread the load. Bedrock may introduce a similar concept that could make use of :ets tables as you describe on each node. For now, calling the Sequencer directly is simple and effective.

There are so many opportunities for optimization in front of us. Currently, the focus is on getting all of the basic operations right, increasing test coverage, getting the documentation right, and working towards getting the system into a place where people might actually dare to use it for real work.

If you see something you want to improve, though, we’re open to a PR!

The way transactions were being assembled in 0.1 left holes in the “reads” and “writes” that could slip through the Resolver if we tried to add range-reads into the mix. This only worked because 0.1 just didn’t do range operations. Take a look at develop, where 0.2 is coming together, though! I’ve recently (yesterday) overhauled the machinery for building transactions (and the transaction format as well) with an eye toward making range operations like range-reads and clears easier to implement, along with other fun possibilities.

Interestingly, in this system, the read-your-writes all happens entirely within the transaction builder. If in a transaction I read key “a”, and get “apple”… and within that transaction write “aardvark” to “a”, then any subsequent read (or range read!) should return the “aardvark” value for “a”. The rest of the system doesn’t see any of this. When a transaction is committed, passes resolution and is pushed to the logs… any subsequent read using that commit version would see the new value for “a”. If that transaction is rolled back, nothing needs to be done or communicated – we just throw the process away.

This is kind of an interesting thing that FDB does, and I shamelessly copied it!

So they have their famous 5s window for transactions, right? This brings us to a question: How do all of the machines involved know when that window advances?

They all have different clocks, and there could be all kinds of skew… so you can’t use local wall-clock time – synchronizing time is a hassle or requires super-expensive equipment (like Spanner). So clock synchronization is out. How do all the servers in the cluster know when that 5s window has advanced?

Here’s what they (and we) do (and I think it is super cool):

• The sequencer issues versions in microsecond increments. This is monotonic time, and so will always be increasing. You can think of it as uptime for the cluster, because versions only advance when the cluster is in a running state. So, all of these servers don’t need to track their own time, they can just look at the most recent version to come along, and subtract ~5 million, and there’s your window. There’s a catch, though: This only works if transactions are happening. If the system is quiescent, nothing happens, nothing advances, the window doesn’t move.
• Commit proxies will issue an empty transaction if no transactions have been processed within the last 1000ms. This advances the version, and slides the window ever-forward. As a bonus, it serves to “tire kick” the transaction system to ensure that the sequencer/resolvers/logs/ etc. are all reachable and functioning. Any error triggers a recovery.

You get this lovely three-for-one out of this: ensuring the parts are working, advancing the window, synchronization.

indeed.

Yep. Broken all over the place. It’s a big project, and it’s just me, and it’s hard to keep all of the references up-to-date as things are in flux. As the design settles down, and the number of people contributing (hopefully) grows, I imagine that things like this will be sorted out.

Yep. The storage server that’s in there now, “Basalt,” is a place-holder, an example. I used :dets in it because it’s simple and it’s built into the BEAM. It could easily be replaced with some other k/v store (like rocksdb), but this works to let people try out the system and do so with minimal fuss, even in a livebook.

One thing interesting thing to point out about this model, though: There doesn’t have to be just one kind of storage server in operation at a time. I could easily see using a mix of them for different properties. Maybe a memory-only one that doesn’t guarantee durability, but offers extremely fast reads? Or another that’s good at absorbing writes quickly (built around an LSM, perhaps), or maybe another that packs transactions and sends them off to AWS/GCS for disaster recovery and can answer reads, but maybe slowly. There are a lot of interesting possibilities here!

I considered this, and might reconsider it again at some point. Ultimately, I found that these patching tools (of which there are a few) don’t seem to play well with asynchronous tests, as different tests might need to patch things in different ways, and yet others may want to use the actual implementation.

I haven’t settled a on a good answer for this… and as you note, there are some small drawbacks to this approach. At this time, though, I think the focus needs to be on correctness. There will come a time, though, that those extra few cycles matter and I’m sure the code will change.

I’ll take that into consideration.

I tend to use a lot of pipelining - that’s the whole story. The macros ensure that the functions are defp, and the compiler will inline them. In the compiled code, there’s no difference between this and hard-coding the tuples, but the functions play nicely with pipelining and i don’t need to resort to then(&{:noreply, &1}).

Yay! I’m really quite happy that this is generating some interest. FDB is an awesome system and the patterns and ideas really should be more widely used. It’s just good tech. Keep the questions and feedback coming!