Hi all! I'm going nuts trying to figure this out, would really appreciate some thoughts from you.
My test suite has 700 tests. Whenever I run a test isolated it works fine. But when the full suite is ran, then I get around 3-6 failing tests. It's not always the same test that fails, and not always with the same values. Things just seem to be quite random! One workaround for this was splitting up the test into groups and run them separately. This worked for a while, but now one particular group is getting large, and I start getting 1-2 random tests failing inside this group.
I can't figure the hell out of this. FYI, I'm running the tests in parallel. If I enable process isolation it seems to help, but then I lose all the benefits of running in parallel. I've tried running with LazilyRefreshDatabase, RefreshDatabase and DatabaseTransactions. Nothing solved it. I can't even wrap my mind around the problem because each process has its own DB so I don't even understand how the heck do values leak from one to the other.
help!
@nunodonato No one can help because we have no idea what your application code looks like, what your tests look like, or what failures you do get because you don’t tell us.
If you want help, you need to provide some actual context and details.
@martinbean I seed data inside (some) tests, I use factories in others. Besides that, just lots of business logic. Sometimes I use transactions, but the failures don't seem to be related with tests that use transactions.
What other context would be helpful to provide that could be related with these failures?
@nunodonato I’ve given you pointers on what to provide (application code, test code, what failures you get) and you still haven’t provided any details of that.
We don’t know your application. We can’t see your screen. You can’t just post “I get test failures sometimes. Why?” and expect completely strangers on the Internet to tell you exactly what the problem is.
If you're using the same database source and it isn't SQLite then you might have race condition. One test override the other.
I finally found a solution. Can't tell you how happy I am, this was a tough nut to crack! I'm leaving here a solution if someone else bumps into this in the future, I hope you don't have to waste so much time as I did https://www.nunodonato.com/solving-randomly-failing-tests-in-laravel/
Please or to participate in this conversation.