volcov

volcov

LetItCrash - A testing library for crash recovery and OTP supervision behavior

Hello,
How is everyone?
I hope you’re well =)

I worked on a project with some legacy code that existed before Oban’s arrival. This legacy code functioned as a “job” that was created alongside dynamic supervisors. Understanding the outcome of a supervisor failure and recovery has always been a challenge.

So I thought about creating a set of tools to help me write tests that would simulate these scenarios.

It’s a narrow path between knowing “if I’m testing the Supervisor” (which isn’t the goal) and whether I’m testing the behavior of the result after the Supervisor has done its job.

And from this thought, this lib was born. I still have many ideas for what to add to it. There are a world of scenarios to cover (Workers, DB connections, State), but version 0.1.0 has the general idea I want to facilitate.

I brought it to the forum to see if the idea makes sense =)
I hope it can help more people who are also facing this challenge.

Hex: let_it_crash | Hex
GitHub:

Most Liked

volcov

volcov

First of all, I apologize for the delay in responding to your comment, I was buried by a big project and I didn’t come back here.

Thank you so much for taking the time to share your thoughts! I completely understand your perspective, and honestly, I questioned myself many times about whether this library was really necessary or if I was just “testing OTP” unnecessarily.

However, after using let_it_crash actively in our production codebase, I found it incredibly valuable for a real-world problem we were facing. Let me share our experience:

Our Use Case

We had a scoring system with this supervision tree:


ScoreSupervisor

├── DynamicSupervisor (manages calculation workers)

└── ScoreCoordinator (coordinates the flow)

The Problem: Our system would occasionally get stuck during the normalization phase in production, but we couldn’t figure out why.

What let_it_crash Revealed

When we wrote tests simulating crashes during active processing:


test crash during active processing" do

ScoreCoordinator.start_score_calculations()  # Start with active workers

LetItCrash.crash!(*ScoreCoordinator*)

  assert LetItCrash.recovered?(ScoreCoordinator)  # ❌ FAILED!

end

We discovered a real bug: Our supervisor used :one_for_all strategy, which:

  1. Killed active workers unnecessarily when coordinator crashed
  2. Created orphaned workers trying to communicate with dead processes
  3. Led to failed recovery and the system getting stuck

What We Were Actually Testing

You’re absolutely right that we shouldn’t test if OTP works (we know it does!). But we were testing:

1. Our Configuration Choices**

  • Is :one_for_all or :rest_for_one better for our specific architecture?

  • Does our coordinator’s trap_exit work correctly with shutdown signals?

2. Our Application Logic

  • Does our cleanup code in terminate/2 properly handle active workers?

  • Is our shutting_down? flag preventing new work correctly?

  • Are monitor references being cleaned up?

3. Interactions Between Components

  • What happens to workers when coordinator crashes mid-calculation?

  • Does state remain consistent after recovery?

  • Can the system continue processing after recovery?

4. Edge Cases Specific to Our Domain

  • Crash with zero entities

  • Crash with exactly @max_concurrent_workers entities

  • Crash during normalization phase

  • Multiple rapid crashes

Conclusion

I initially had the samequestion you expressed. But when our production system had a mysterious bug that only happened under specific failure conditions, let_it_crash gave us a systematic way to:

  1. Reproduce the issue in tests

  2. Understand the root cause

  3. Validate the fix

  4. Ensure it doesn’t regress

I don’t know if I managed to express exactly what this journey was like, let me know if I left more questions than answers haha

Thanks again for the discussion :purple_heart:

Eiji

Eiji

While usually there is no rule that’s always right and there are almost always edge cases we as a community still try to make conventions clear and simple for everyone, so in some specific cases we know what to expect. It’s not rather about how to life, but something like a standard explaining quality of life.|

Therefore it would be amazing especially for a new developers if you could write the code that is compatible with our naming convention. Here is a direct link to the section with a trailing bang character in the function name: Trailing bang (foo!) | Naming conventions @ Elixir hex documentation

Eiji

Eiji

I think that the crash! naming may be very confusing …

Unlike crash/1, this function uses :kill instead of :shutdown, which ensures the process is terminated even if it has Process.flag(:trap_exit, true). This is useful for testing processes that trap exits, such as GenServers that need to perform cleanup on normal exits.

I would change it to something like:

@spec crash(type :: :shutdown | :kill, pid())
def crash(type \\ :shutdown, piid)
volcov

volcov

I was thinking about doing it the way you suggested before going down the ! route, but I thought it would be “less verbose” by just adding the exclamation point referencing “critical operation” haha.

Thinking twice, this might cause confusion about the actual meaning. I have other changes coming soon, and I’ll add this one as well.

Thank you so much for the feedback :purple_heart:

Where Next?

Popular in Announcing Top

rodloboz
I’ve started working on a new library to run SQL queries and do basic business intelligence. Think “Blazer for Elixir.” Currently it fe...
New
wkirschbaum
Hi everyone. We released Spreadsheet a couple of months ago which we have been using internally for parsing xlsx files internally for our...
New
zteln
Hello Elixir community, I’ve written a small embedded key-value database in Elixir called Goblin. It’s based on the Log-Structured Merge...
New
fuelen
Hi all! Confispex is a tool which allows defining specs for runtime configuration, cast values according to specified types and inspect ...
New
bentanweihao
Here’s a preview: Here’s the link to download: GenServer cheatsheet Tell me what you think!
New
murrgelb
Efx is a library to define and test side effects declaratively. It is basically a very focused mocking framework, reducing implementation...
New
martosaur
Hello, I’m excited to introduce InstructorLite – a fork of the Instructor package. Instructor brought the very idea of structured LLM p...
New
Asd
Hi, I am happy to release the Repatch library for mocking and patching implementation in tests and anywhere else. It brings new possibili...
New
sevensidedmarble
Announcing Live Toast: a replacement toast/flash component for Phoenix LiveView, heavily inspired by the look of Sonner (the amazing toas...
New
fuelen
Hey folks! Want to present a toolkit for writing command-line user interfaces. It provides a convenient interface for colorizing text...
New

Other popular topics Top

srinivasu
How to handle excepions in elixir? Suppose i have A, B, C ,D, E modules. and each module has get() function. A.get() method will call th...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
joaquinalcerro
Hi there, I am working with Ecto-Postgresql and I need to call all of the records from a specific table but the table has 40,000 record...
New
Jim
As a follow up to my earlier question: I have the code compiling and running but not getting a successful login from the rest server. ...
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
TunkShif
This post is an instruction guide to help you setup your Neovim for Elixir development from scratch. It includes general information on h...
273 38985 115
New
AstonJ
We’ve put together this wiki for Phoenix LiveView - please feel free to add any info you feel is worth including. What is Phoenix LiveV...
New
magnetic
Hey :wave:t3: Elixir community, I’ve been learning Elixir, and working on some side projects. My editor of choice is VSCode, and althoug...
New

We're in Beta

About us Mission Statement