keliumJU
A single test passes, but if I run the whole test suite, the same test fails
Hello everyone, lately I have encountered this problem, I have seen it happen in several languages, any idea how to deal with it in elixir?
For example, when I run
mix test test/app_web/live/module/index_live_test.exs → pass
but if I do
mix test → only the above test fails
I want to clarify that in my case I use Task.async in that module and in the test I also use Task.async and send() to send those tasks.
P.S: Actually, this works but separating the problematic tests to another test module, I don’t know if this is considered bad practice.
P.S2: Sometimes it was solved with Process.sleep(300) or adding async: true to the ConnCase config.
Thank you very much in advance.
Marked As Solved
keliumJU
Yes I would think they are side effects, because the error it generates can be described as: “it is not reached to send the Task.async therefore no such tasks exist and the test fails”, I have separated the tests that handle Task.async to another describe with a different setup and that works for now for me, thank you very much for the general context of the problem.
P.S: I have not moved any of the test configuration (rollback question).
Also Liked
sodapopcan
It’s impossible to know without seeing your code but this generally happens because one test is causing persisting side effects that another test is reading. In the past this was usually the database but Phoenix, and most other frameworks, rollback the db state on each test. Did you turn that off, per chance? Another possibility is that one test is writing a file that another is expecting not to be there. There are a lot of different possibilities but that’s the gist of it.
iarekk
General overview: your code has mysterious side-effects that make the tests flaky.
Potential steps to investigate:
- Isolate the tests into a reproducible and publishable piece of code.
- Hire a consultant, have them sign and NDA and debug it in your codebase.
I don’t think a productive discussion can be had with the details you’ve been able to provide so far.
keliumJU
Yes, sorry, ideally I would like to have a project with the reproducible bug, right now I solved it by using a different describe with a different setup for the problematic tests (in same file), in a future post I will be sure to follow the steps you recommend, thank you very much.
Popular in Questions
Other popular topics
Latest Phoenix Threads
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Develop your skills with books, videos, and courses.
Producing high quality Elixir screencasts since 2017.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"