volcov

volcov

LetItCrash - A testing library for crash recovery and OTP supervision behavior

Hello,
How is everyone?
I hope you’re well =)

I worked on a project with some legacy code that existed before Oban’s arrival. This legacy code functioned as a “job” that was created alongside dynamic supervisors. Understanding the outcome of a supervisor failure and recovery has always been a challenge.

So I thought about creating a set of tools to help me write tests that would simulate these scenarios.

It’s a narrow path between knowing “if I’m testing the Supervisor” (which isn’t the goal) and whether I’m testing the behavior of the result after the Supervisor has done its job.

And from this thought, this lib was born. I still have many ideas for what to add to it. There are a world of scenarios to cover (Workers, DB connections, State), but version 0.1.0 has the general idea I want to facilitate.

I brought it to the forum to see if the idea makes sense =)
I hope it can help more people who are also facing this challenge.

Hex: let_it_crash | Hex
GitHub:

Most Liked

volcov

volcov

First of all, I apologize for the delay in responding to your comment, I was buried by a big project and I didn’t come back here.

Thank you so much for taking the time to share your thoughts! I completely understand your perspective, and honestly, I questioned myself many times about whether this library was really necessary or if I was just “testing OTP” unnecessarily.

However, after using let_it_crash actively in our production codebase, I found it incredibly valuable for a real-world problem we were facing. Let me share our experience:

Our Use Case

We had a scoring system with this supervision tree:


ScoreSupervisor

├── DynamicSupervisor (manages calculation workers)

└── ScoreCoordinator (coordinates the flow)

The Problem: Our system would occasionally get stuck during the normalization phase in production, but we couldn’t figure out why.

What let_it_crash Revealed

When we wrote tests simulating crashes during active processing:


test crash during active processing" do

ScoreCoordinator.start_score_calculations()  # Start with active workers

LetItCrash.crash!(*ScoreCoordinator*)

  assert LetItCrash.recovered?(ScoreCoordinator)  # ❌ FAILED!

end

We discovered a real bug: Our supervisor used :one_for_all strategy, which:

  1. Killed active workers unnecessarily when coordinator crashed
  2. Created orphaned workers trying to communicate with dead processes
  3. Led to failed recovery and the system getting stuck

What We Were Actually Testing

You’re absolutely right that we shouldn’t test if OTP works (we know it does!). But we were testing:

1. Our Configuration Choices**

  • Is :one_for_all or :rest_for_one better for our specific architecture?

  • Does our coordinator’s trap_exit work correctly with shutdown signals?

2. Our Application Logic

  • Does our cleanup code in terminate/2 properly handle active workers?

  • Is our shutting_down? flag preventing new work correctly?

  • Are monitor references being cleaned up?

3. Interactions Between Components

  • What happens to workers when coordinator crashes mid-calculation?

  • Does state remain consistent after recovery?

  • Can the system continue processing after recovery?

4. Edge Cases Specific to Our Domain

  • Crash with zero entities

  • Crash with exactly @max_concurrent_workers entities

  • Crash during normalization phase

  • Multiple rapid crashes

Conclusion

I initially had the samequestion you expressed. But when our production system had a mysterious bug that only happened under specific failure conditions, let_it_crash gave us a systematic way to:

  1. Reproduce the issue in tests

  2. Understand the root cause

  3. Validate the fix

  4. Ensure it doesn’t regress

I don’t know if I managed to express exactly what this journey was like, let me know if I left more questions than answers haha

Thanks again for the discussion :purple_heart:

Eiji

Eiji

While usually there is no rule that’s always right and there are almost always edge cases we as a community still try to make conventions clear and simple for everyone, so in some specific cases we know what to expect. It’s not rather about how to life, but something like a standard explaining quality of life.|

Therefore it would be amazing especially for a new developers if you could write the code that is compatible with our naming convention. Here is a direct link to the section with a trailing bang character in the function name: Trailing bang (foo!) | Naming conventions @ Elixir hex documentation

Eiji

Eiji

I think that the crash! naming may be very confusing …

Unlike crash/1, this function uses :kill instead of :shutdown, which ensures the process is terminated even if it has Process.flag(:trap_exit, true). This is useful for testing processes that trap exits, such as GenServers that need to perform cleanup on normal exits.

I would change it to something like:

@spec crash(type :: :shutdown | :kill, pid())
def crash(type \\ :shutdown, piid)
volcov

volcov

I was thinking about doing it the way you suggested before going down the ! route, but I thought it would be “less verbose” by just adding the exclamation point referencing “critical operation” haha.

Thinking twice, this might cause confusion about the actual meaning. I have other changes coming soon, and I’ll add this one as well.

Thank you so much for the feedback :purple_heart:

Where Next?

Popular in Announcing Top

metehan
ExTerm is a minimal Elixir library that lets you expose your terminal in a web browser. It’s mostly a toy project for learning and experi...
New
kzemek
I’ve recently released v0.2.0 of my Python interop library, Snex. This version rolls up all work-in-progress improvements that have been...
New
New
mindreframer
ElixirProto: Protobuf-Inspired Serialization for Elixir Events I wanted to have an Elixir-native serialization for events, that also supp...
New
hauleth
It is library created by me and @abc3. It is simple client for DuckDB, but with small twist when compared with other libraries out there ...
New
lud
Hello! I’ve been working on the Oaskit library for a while now, and just released a first version. Since I’ve built JSV I wanted to be ...
New
lostbean
Hello Elixir community! :wave: I’m excited to announce the first release of ReqCassette, a VCR-style record-and-replay library specifica...
New
jstimps
I’ve started development on an Ecto Adapter for FoundationDB: GitHub - ecto_foundationdb. FoundationDB is a distributed database with AC...
New
corka149
A JSON patch is a way to define a sequence of manipulating operations on a JavaScript object. The IETF published the RFC 6902 - found he...
New
fhunleth
Elixir Circuits is a set of libraries for interacting with hardware. We previously announced Circuits.UART, and now we’re ready to announ...
New

Other popular topics Top

shahryarjb
Hello, I get Persian date from my client and convert it to normal calendar like this: def jalali_string_to_miladi_english_number(persi...
New
peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New

We're in Beta

About us Mission Statement