code
Why is table fragmentation not possible with local_content tables in mnesia?
mnesia disc copies have a size limit of 4Gb, a solution to this is to fragment the table, however, this feature is not available to local_content tables, why is that?
And is there a solution other than implementing a feature that checks the size of the current table and create a new one when it approaches the size limit, which will consequently lead to implementing another feature for querying multiple tables when a record from that table is needed.
It seems like mnesia is starting to get a bit outdated because of dets. On the other hand, I don’t want to use something like leveldb as it’s not a part of the ecosystem. What would you do?
Thank you.
Marked As Solved
garrison
I don’t know the answer to your question. I tried a bit of git blaming to see when that line in the docs was added and it goes back at least 16 years (probably older), so I wonder if anyone even remembers 
I think this is the understatement of the century lol
Probably use SQLite. It’s well-adopted here (Exqlite) and very stable.
For the record, I have been slowly progressing on an Elixir DB project but it goes without saying that it’s very tricky. I will be using SQLite as the underlying (disk) storage engine, at least for a while. Writing a storage engine is not easy.
There are a couple more beam-native K/V stores, though. There’s leveled built as a pure-Erlang storage engine for Riak. It’s an LSM so no good for sequential reads.
There’s CubDB which is an on-disk COW tree. It’s not a very advanced cowtree (no incremental GC), so probably no good for “serious” usage, but it might get the job done.
Finally there’s Khepri which is a raft-replicated in-memory (but also persisted) K/V store, which I think is intended as an Mnesia successor. I don’t know what sort of limits they have for table size, but you would be limited by your RAM at least.
Also Liked
garrison
Still not ready to talk about it yet, but the goal is high availability, durability (replication), horizontal scalability, and strict serializability (external consistency) by default with an ordered K/V model (ordered is very important), in pure Elixir (with SQLite for a while). I am now pretty confident I can actually meet those requirements, though performance remains an open question (things have been looking acceptable to me so far).
I then intend to layer a relational database on top of it so that I can finally escape Postgres and ascend into the BEAM, or something.
dimitarvp
Any more details? What problem will it solve?
dimitarvp
It’s a delicate balance in expression when you want to be both revolutionary and an experienced techie at the same time – and that guy didn’t nail it, that much is certain.
But the core of his argument is sound: everyone just immediately reaches for algorithms A and B when it comes to K/V stores, and algorithms C, D and E for distribution etc.
He’s right that uncritically reaching for stuff and just hoping you are the one who assembles a LEGO juuuust a smidgen better than the 561 guys before you is… not optimal. And not productive.
So I appreciate him for being a bit of a revolutionary. We’ll see though, some 6 months down the line, after the whole thing marinates a little bit.
Sure, but the BEAM ecosystem has the drawback of most stuff being ephemeral. As the BEAM ecosystem – and all others – mature and try to converge into more stable and less “reinvent the wheel every time” approach, I believe persistent should become a default.
(And I realize you were talking about caches. I don’t do, I mean just K/V stores and message queues in general.)
dimitarvp
You are not missing anything, I simply mixed expressing what I would prefer when needing K/V stores or MQ software, with your criticism towards the README of nubmq.
Apologies.
As for the rest of your remarks, I don’t disagree. I appreciate the guy for starting from scratch. If he ended up rediscovering old wisdom that’s also a very valid outcome.
legoscia
mnesia disc copies have a size limit of 4Gb
No, disc_copies tables are only limited by the amount of available RAM (on 64-bit systems, at least).
disc_only_copies tables are implemented using Dets tables, and thus limited to 2 GB. disc_copies tables used to use Dets tables as well, but that was changed in Erlang/OTP R7B-4.
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Courses that'll move you from confusion to "Aha, now I get it!"
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Courses that'll move you from confusion to "Aha, now I get it!"