I’ve worked on multiple systems that democratized their data through kafka and other mechanisms. Based on those anecdotes I’m more then comfortable suggesting that most companies should not do this.

All that aside, most companies are information systems. Meaning they take data, store data, and present that data to users as information. That being the case the database absolutely does matter. Its not “just a detail”. Its an integral part of your business and you should choose databases that have the tradeoffs your business needs.

IMHO If the purpose of the application is to store and manage data, then you can either use a database or build a database, and you probably don’t want to build a database.

However, there are applications who have jobs that isn’t storing and managing data. For example, we have applications which route data and others that serve as kiosk systems. For these, a database was superfluous. Just my $0.02.

I recently tried this approach building a new production service. It managed lists of contacts with some additional business logic, each list getting its own GenServer with state backed up to S3. Each GenServer would gracefully shut down after a few minutes of inactivity to avoid memory leaks. In short, we scrapped the whole thing and reimplemented with postgres.

Building without a database allows you to write some of the most expressive code you’ll ever create, at the expense of having to write a lot of things you take for granted in traditional design.

Big issues I faced:

• Needing sagas almost immediately. Simple pieces of information had be duplicated in a few places, and these updates had to be atomic.
• Data migrations. My initial version directly wrote the struct with term_to_binary, but obviously this gets hairy if data needs to change. Lacking the time to implement a proper migration strategy due to deadlines, I ultimately decided to abandon the whole approach.

It’s entirely possible I implemented it wrong, and I do intend to try it again in the future. Ultimately I felt like the system I had built wasn’t nearly as stable or simple as a database, and given the scheduling deadlines, stability and simplicity had to come first.

As a side note, the original business requirements had us under the impression that contact lists would be no greater than 10K entries. We had a 45K list day one in production, with the expectation to have multiple 100K+ lists in the coming weeks.

Everything depends on the use case.
I am pretty sure you have some concrete examples in mind while asking. It might be useful to share your particular scenario.

In this book Functional Web Development with Elixir, OTP, and Phoenix author shows how to make an interactive game. When the game is ready and works in memory, he adds saving state in ETS in case of crashes and mentions that it could use dets (disk based term storage).

He does exactly the thing you describe as confusing: starts without touching the DB and ends up with an excellent model decoupled from storage. I highly encourage reading it!

[EDIT] I would also treat Why Relational Databases Are So Bad with a grain of salt. The timesheet example might ring a bell in people. Adding sequence ids to rows in data and work to produce a report from SQL is repetitive. The same goes with retrieving employee name from a different table.

What the author doesn’t mention is that storing employees and hours separtely and normalising the database allows to easily reverse the query: “who worked at a given time?” Try that with saving timesheets as a list of hours in a file

Again: everything depends on the use case. Maybe you will never need that reversed query?

This is exactly right. Unless your company sells a database don’t build a database.

To add on to this, I don’t know of many companies that don’t need to store something. My advice is to put your state into a reliable database (postgres is a good default). As other people have said this empowers reporting, etl, and a whole bunch of other benefits.

It’s not the only reason, but the main reason to build stateful systems - meaning bringing your application’s state into processes - is to reduce latency. I have a lot of empirical evidence to support that a “stateless” elixir service backed by a database will take you a really long way. IMO you need to prove that postgres or your db of choice won’t be fast enough before you start bringing more state into your application.