This is very interesting, and I’ve been thinking of doing the same. From a cursory reading of the code, it looks like you mutate all operators in the file at the same time instead of one by one. Is that so? Shouldn’t one mutate one operator at the time to get more precise results?

To reduce the need to recompile a module lots of times, I’ve also tried to do it in a different way by mutating the Erlang AST in a way that allowed me to toggle mutations on and off without recompiling the code (see here: https://nwolverson.uk/devlog/2016/08/01/introducing-purescript-erlang.html), but I couldn’t get very good error messages on my first try. It also made it harder to generate relevant mutations (and it made it impossible to generate mutations on macros)

When I tried to do it it didn’t seem as easy as you might think. I tried to traverse the AST while keeping a counter, so that I knew which operator to mutate, but I didn’t manage to make it work. I must have been doing something wrong.

Yes, that’s the main reason to mutate one at a time.

Recompiling your modules will in general be much slower than running the tests, so I think you’re optimizing for the right thing (i.e., reducing the need to recompile code, at the cost of possibly running more tests)

Because of what I said above, I don’t think you should even try to guess which tests to run…

I’ve just found out that recompiling modules lots of times (~ hundreds of times), even if the modules don’t change consumes memory from the BEAM’s literal allocator. This memory doesn’t seem to be garbage collected and it will fill up and cause errors. If you decide to generate mutants one at a time and each mutant requires a module recompilation, you might hit these errors…

I’m just warning you because I found out about this problem the hard way (i.e., when I tried to recompile my test suite about 300 times for 300 mutations in an elixir module). Fortunately, the way I’m generating mutants doesn’t require recompiling the code for each mutant, so I could work around it.

I’ll look into it. IMO, if we are recompiling the same module, we shouldn’t consume new memory from the literal allocator. And I did try to clean up the code as much as I could. I’m not as motivated to get at the bottom of this, though, because my approach no longer depends on recompiling modules…

Hey Luiz, thanks for your interest in exavier. For now I think you should be able to overcome that by setting EXAVIER_DEBUG=1.

MRs welcome to allow setting custom timeouts for each particular time-bound work.