You’re right that Muzak isn’t as supported as I’d like it to be (for reasons), but I am releasing a new version this weekend that works for 1.14, and should hopefully work for all future Elixir versions. I’m hopeful that I’ve gotten it off of any private APIs at this point.

Yes, the situation that you describe where you can essentially delete the “code under test” entirely and the tests still pass is basically the canonical example of what mutation testing helps with.

Indeed, mutation testing is property testing for your tests

Release compatible with 1.14 is up here: muzak | Hex

More info on Muzak Pro is here.

The biggest thing you get is git integration so you can effectively include mutation testing as part of your CI process if you would like. It restricts the mutations generated to only the lines that have changed since the last merge commit. If you’re only generating mutations for the LOC that have changed, though, you can run mutation testing on that and it shouldn’t add too much time to your CI runtimes. You can also define a custom percentage that you’re looking to hit to indicate “success” for the run in the configuration to work how your team wants in CI.

And of course, because the runtimes of mutation testing increase as your number of surviving mutants increases, as your test suites become better the time spent in mutation testing goes down!

I don’t think mutation testing is necessarily going to help much given the problems you’ve described - the changes it makes (random example: ROR3 from Exavier) are mostly function-scale. That kind of mutation is good for making sure your tests cover the < and the = situations for an <= comparison, but “we removed a whole module and the tests still passed” is a bigger issue.

My interpretation of your situation is that the codebase has fallen into a mocking trap; there are real implementations that are replaced with a mock EVERYWHERE and not tested individually. Some ways to address that:

• write specific tests for the thing that’s being mocked everywhere. Ideally there would be a corresponding test for every scenario that’s set up in the mocks, to demonstrate that the real thing actually does the what the mocks are pretending to do.

• write higher-level integration tests that don’t use mocks. For a legacy codebase, the “happy path” is a good place to start. These will be slower than isolated unit tests, so you may want to tag them and run them as a separate CI step.