beardedeagle

beardedeagle

Mnesia crashing on nodeup in clustered configuration

I am running into a problem where mnesia is crashing on nodeup in clustered mode:

Mnesia('vanguard@test02'): mnesia_monitor starting: <0.259.0>
Mnesia('vanguard@test02'): Mnesia debug level set to verbose
Mnesia('vanguard@test02'): mnesia_subscr starting: <0.260.0>
Mnesia('vanguard@test02'): mnesia_locker starting: <0.261.0>
Mnesia('vanguard@test02'): mnesia_recover starting: <0.262.0>
Mnesia('vanguard@test02'): mnesia_tm starting: <0.263.0>
Mnesia('vanguard@test02'): Schema initiated from: default
Mnesia('vanguard@test02'): mnesia_controller starting: <0.265.0>
Mnesia('vanguard@test02'): mnesia_late_loader starting: <0.266.0>
Mnesia('vanguard@test02'): Intend to load tables: []
Mnesia('vanguard@test02'): mnesia_controller terminated: shutdown
Mnesia('vanguard@test02'): mnesia_tm terminated: shutdown
Mnesia('vanguard@test02'): mnesia_recover terminated: shutdown
Mnesia('vanguard@test02'): mnesia_locker terminated: shutdown
Mnesia('vanguard@test02'): mnesia_subscr terminated: shutdown
Mnesia('vanguard@test02'): mnesia_monitor terminated: shutdown
Mnesia('vanguard@test02'): mnesia_monitor starting: <0.280.0>
Mnesia('vanguard@test02'): Mnesia debug level set to verbose
Mnesia('vanguard@test02'): mnesia_subscr starting: <0.281.0>
Mnesia('vanguard@test02'): mnesia_locker starting: <0.282.0>
Mnesia('vanguard@test02'): mnesia_recover starting: <0.283.0>
Mnesia('vanguard@test02'): mnesia_tm starting: <0.284.0>
Mnesia('vanguard@test02'): Schema initiated from: default
Mnesia('vanguard@test02'): mnesia_controller starting: <0.286.0>
Mnesia('vanguard@test02'): mnesia_late_loader starting: <0.287.0>
Mnesia('vanguard@test02'): Intend to load tables: []
Mnesia('vanguard@test02'): Mnesia started, 0 seconds
[info] Application mnesia exited: :stopped
Mnesia('vanguard@test02'): Logging mnesia_up 'vanguard@test01'
Mnesia('vanguard@test02'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test02'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.StatusStore' 4 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('vanguard@test02'): Creating index for 'Elixir.CheckStore' 10 ordered

If I kill node 1, wait, then bring it back up I get the following:

Mnesia('vanguard@test01'): mnesia_monitor starting: <0.263.0>
Mnesia('vanguard@test01'): Mnesia debug level set to verbose
Mnesia('vanguard@test01'): mnesia_subscr starting: <0.264.0>
Mnesia('vanguard@test01'): mnesia_locker starting: <0.265.0>
Mnesia('vanguard@test01'): mnesia_recover starting: <0.266.0>
Mnesia('vanguard@test01'): mnesia_tm starting: <0.267.0>
Mnesia('vanguard@test01'): Schema initiated from: disc
Mnesia('vanguard@test01'): Initial dump of log during startup: [dumped,
                                                                               dumped]
Mnesia('vanguard@test01'): mnesia_controller starting: <0.290.0>
Mnesia('vanguard@test01'): mnesia_late_loader starting: <0.291.0>
Mnesia('vanguard@test01'): Intend to load tables: [{'Elixir.ProcStore',
                                                                   ram_only}]
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test03'
Mnesia('vanguard@test01'): mnesia_controller terminated: shutdown
Mnesia('vanguard@test01'): mnesia_tm terminated: shutdown
Mnesia('vanguard@test01'): mnesia_recover terminated: shutdown
Mnesia('vanguard@test01'): mnesia_locker terminated: shutdown
Mnesia('vanguard@test01'): mnesia_subscr terminated: shutdown
Mnesia('vanguard@test01'): mnesia_monitor terminated: shutdown
Mnesia('vanguard@test01'): mnesia_monitor starting: <0.319.0>
Mnesia('vanguard@test01'): Mnesia debug level set to verbose
Mnesia('vanguard@test01'): mnesia_subscr starting: <0.320.0>
Mnesia('vanguard@test01'): mnesia_locker starting: <0.321.0>
Mnesia('vanguard@test01'): mnesia_recover starting: <0.322.0>
Mnesia('vanguard@test01'): mnesia_tm starting: <0.323.0>
Mnesia('vanguard@test01'): Schema initiated from: default
Mnesia('vanguard@test01'): mnesia_controller starting: <0.325.0>
Mnesia('vanguard@test01'): mnesia_late_loader starting: <0.326.0>
Mnesia('vanguard@test01'): Intend to load tables: []
Mnesia('vanguard@test01'): Mnesia started, 0 seconds
[error] Error in process #PID<0.295.0> on node :"vanguard@test01" with exit value:
{:badarg, [{:ets, :insert, [:mnesia_gvar, {:last_error, {{:aborted, {:node_not_running, :"vanguard@test01"}}, [{:mnesia_loader, :wait_on_load_complete, 1, [file: 'mnesia_loader.erl', line: 358]}, {:mnesia_tm, :apply_fun, 3, [file: 'mnesia_tm.erl', line: 836]}, {:mnesia_tm, :execute_transaction, 5, [file: 'mnesia_tm.erl', line: 811]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}}], []}, {:mnesia_lib, :set, 2, [file: 'mnesia_lib.erl', line: 444]}, {:mnesia_lib, :fix_error, 1, [file: 'mnesia_lib.erl', line: 906]}, {:mnesia_tm, :return_abort, 3, [file: 'mnesia_tm.erl', line: 958]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}

[error] Error in process #PID<0.294.0> on node :"vanguard@test01" with exit value:
{:badarg, [{:ets, :insert, [:mnesia_gvar, {:last_error, {{:aborted, {:node_not_running, :"vanguard@test01"}}, [{:mnesia_loader, :wait_on_load_complete, 1, [file: 'mnesia_loader.erl', line: 358]}, {:mnesia_tm, :apply_fun, 3, [file: 'mnesia_tm.erl', line: 836]}, {:mnesia_tm, :execute_transaction, 5, [file: 'mnesia_tm.erl', line: 811]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}}], []}, {:mnesia_lib, :set, 2, [file: 'mnesia_lib.erl', line: 444]}, {:mnesia_lib, :fix_error, 1, [file: 'mnesia_lib.erl', line: 906]}, {:mnesia_tm, :return_abort, 3, [file: 'mnesia_tm.erl', line: 958]}, {:mnesia_loader, :init_receiver, 5, [file: 'mnesia_loader.erl', line: 284]}, {:mnesia_loader, :do_get_network_copy, 5, [file: 'mnesia_loader.erl', line: 220]}, {:mnesia_controller, :"-load_table_fun/1-fun-3-", 5, [file: 'mnesia_controller.erl', line: 2180]}, {:mnesia_controller, :"-load_and_reply/2-fun-0-", 2, [file: 'mnesia_controller.erl', line: 2138]}]}

[info] Application mnesia exited: :stopped
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test03'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test02'
Mnesia('vanguard@test01'): Logging mnesia_up 'vanguard@test01'
Mnesia('vanguard@test01'): Creating index for 'Elixir.ProcStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 4 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.CheckStore' 10 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.HandlerStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.HandlerStore' 7 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.StatusStore' 3 ordered
Mnesia('vanguard@test01'): Creating index for 'Elixir.StatusStore' 4 ordered

I’m not really able to make heads or tails of the situation unfortunately. I am using libcluster, Swarm and Mnesiam for distributed libs. Any and all help/tips/refrences are appreciated.

Most Liked

beardedeagle

beardedeagle

Ok, I’ve since forked mnesiam here (with the blessing of the original lib author):

And i’ve documented the issue here to work on it as I have time:

Where Next?

Popular in Questions Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
logicmason
Hi there, I'm working through my first release with elixir/phoenix. I've built a release with distillery and found that it crashes when I...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
nsuchy
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
rms.mrcs
Hi, I need to transform a list of numbers into a map where the keys are the indexes and the values are the original values of the list....
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New

We're in Beta

About us Mission Statement