Inspired by some of the discussions on this thread, I decided to play around with different failure modes with mnesia distribution: https://github.com/keathley/breaking_mnesia. I have a large test file that tries to explain my thought process. A cynical person could probably claim that these “failures” are either “working as intended” or “operator error”. My point wasn’t to try to say that mnesia is broken. My goal was to demonstrate the ways that Mnesia might be surprising if you’re expecting it to provide certain guarantees.

You can make some simple API so you can access the app from the outside using curl or something simple. You need 2 API access:

1. check data integrity, return ok or not
2. do some random read write

Then you can make a shell script with an infinite loop:

1. mix phy.server
2. check integrity. If failed, stop right here
3. random access
4. kill -9

Then you can run it over night to see if you can trigger something.

The first thing that comes to mind is the node name changed, so the schema path changed. Like running
iex -S mix
then running
iex --sname test -S mix

The idea behind Mnesia is you MUST run it in at least 2 node configuration, as all writes go to both nodes, if 1 node has a power problem / CTRL+C+C, the other node will still receive the writes. (So 3 nodes total, your app + 2 noded mnesia). If you trip over the powercord to the rack hosting both nodes you are screwed and will lose (alot of) data.

Mnesia has no writeaheadlog like (all?) modern databases, it uses a log but its just an in-ram log that gets periodically flushed to disk. (This is why rocks_db backend for Mnesia is not a real solution to me, it doesnt add a WAL)

There was ways around it, you can set the logdumpthreshold to 1 millisecond, so mnesia will dump the log every 1ms. Or set it to 1 write (default is 1000 i think). But dumping this log is very cpu intensive and your tables will lock up as the dump is done, large tables will be rendered unwritable.

If you want to run Mnesia in a single local node configuration (just to store some persistent state for your app in a simple way), id run it with logdumpthreshold 1 write.

To me it sounds like a new set of schema was created so make sure that:

• node name has not changed
• The path to the mnesia files have not changed (look for :mnesia configuration options)

In case the path and node name are correct you can check the status of mnesia itself:

You can check the status of mnesia with :mnesia.system_info(), :mnesia.info(). See man pages for mnesia for more information, or perhaps this page: https://erlang.org/doc/apps/mnesia/Mnesia_chap7.html

You can look for an mnesia core dump to figure out if something crashed. They are called MnesiaCore.<nodename>.<when>.

A quick comment on your startup code. You should generally only create the schema and table once, not every startup. This is also mentioned in the Memento documentation. I don’t know if that has anything to do with your problem (likely not) but something you should consider before moving into production.

Well, it wasn’t necessarily you who changed it so don’t write this off until you can confirm it is not the problem.

As an example from the memento configuration:

# config/config.exs
config :mnesia,
  dir: '.mnesia/#{Mix.env}/#{node()}'        # Notice the single quotes

Which means if you change node name or your mix env you will end up with a different directory.

I don’t know if you followed this configuration but that could give an idea why something would change.

Regarding :mnesia.system_info I would probably just check that the directory is what you expect, that the node is started and accessible and that the tables are of the type you expect.

Interactive Elixir (1.10.1) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Memento.stop

14:34:19.460 [info]  Application mnesia exited: :stopped
:ok
iex(2)> Memento.Schema.create([node()])
:ok
iex(3)> Memento.start()
:ok
iex(5)> :mnesia.system_info()
===> System info in version "4.16.1", debug level = none <===
opt_disc. Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used.
use fallback at restart = false
running db nodes   = [nonode@nohost]
stopped db nodes   = []
master node tables = []
remote             = []
ram_copies         = []
disc_copies        = [schema]
disc_only_copies   = []
[{nonode@nohost,disc_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []

Gives you some information. Especially the Directory "/home/martink/mnesia/what/Mnesia.nonode@nohost" is used line shows you where you store the mnesia datafiles.

You can also double-check that the node is started and that the tables are in fact in :disc_copies mode.

If you search for schema.Dat you might find others mnesia schemas being created.

I don’t think so but perhaps if there is some funny race condition? Normally mnesia will detect that there is a database there already and give you and already_exists error.