Very slow PHPUnit tests using PHP7.2 or PHP7.1.

Published 8 months ago by mstnorris

If I run my tests using PHP7.2 or PHP7.1 they are about 3x slower than if I run them using PHP7.0. Is there anyway to get to the bottom of why this is happening?

I'm using Laravel 5.5.20 and Laravel Homestead 7.0.1. I have 47 rather simple tests, some hitting the database, others just simple assertions; so there isn't anything that should take ages.

I installed johnkary/phpunit-speedtrap to see which tests take the longest so I could remove those but there isn't a specific test that takes a long time because if I remove the offending test, the next one will take ages (see below).

First Run                Second Run
Test A    0.2 sec        Test A    0.2 sec
Test B.   0.3 sec        Test B.   0.3 sec
Test C    0.1 sec        Test C    0.1 sec
Test D    0.1 sec        Test D    0.1 sec
Test E    9.3 sec        REMOVED Test E
Test F    0.3 sec        Test F    9.3 sec <-- Test F now takes ages
Test G    0.2 sec        Test G    0.2 sec

I am also using an in-memory SQLite3 database, with the Laravel CreatesApplication and RefreshDatabase trait as I want each test to run independently.

I do not have Xdebug installed or running. Is there something known that PHP7.1 and PHP7.2 take a long time to run PHPUnit tests? Is there something else I can install (or even run it with Xdebug) to track down what exactly it is that is causing the issue?

Setup

Laravel 5.5.20
Laravel Homestead 7.0.1 (Per-project installation)
PHPUnit 6.4.4
Vagrant 2.0.1
Virtualbox 5.2.4

Results

PHP 7.2 PHPUnit 6.4.4
Time: 12.4 seconds, Memory: 162.00MB

PHP 7.1 PHPUnit 6.4.4
Time: 12.19 seconds, Memory: 162.00MB

PHP 7.0 PHPUnit 6.4.4
Time: 4.88 seconds, Memory: 162.00MB
MikeHopley

What happens if you remove Test D? I'm wondering whether that test is somehow causing expensive setup for the following test.

One general suggestion:

You can split your tests into separate suites, such as unit and feature. The unit suite would be used for tests that do not hit a database, so you can dispense with database setup/teardown.

As well as making your tests faster, it can be helpful when tests fail. If your unit suite passes but your feature suite has many failures, then you immediately know it's a database issue.

mstnorris

Thanks, all my tests are split up into various different groups and suites so I can run a smaller subset. This only happens when they are run together and I as you suggested (which I have done many times) I can remove the offending test, and/or the test that's run before it, and unfortunately it makes no difference.

MikeHopley

Interesting.

Does that mean you only get the slow-down when you are running all the suites together? What about if you run each suite separately?

It's very odd. :(

mstnorris

I get the slow down when I run them (the test suites) separately too. The issue does appear to be with PHP7.1 and PHP7.2 as the tests run very fast when using PHP7.0.

If I run the tests individually however, they do run pretty quickly regardless of the version of PHP.

MikeHopley

Presumably that means you're getting slow-down even when just running your unit suite, which I assume does not involve any database setup / teardown.

If so, at least you've eliminated that possibility.

No idea why the tests would be running slower on newer PHP, sorry. :(

shez1983

what if you re-arranged the tests so the first few happen afterwards? or run that test on its own to see how long it takes in isolation?

i know pretty stupid suggestions but it is truly baffling..

ONe other trick that i always do now is have a mysql testing db - which is seeded and in my tests i use TransactionTrait so each test DOES run in isolation... without having to refresh db in each test..

mstnorris

Rearranging the tests doesn't matter as it isn't a specific test that takes the time, it is always let's say the 10th one to run. I will dig a little deeper and check that this is the case all the time and that it really doesn't matter what the order is.

I have run the tests in isolation and independently and they're very fast. The issue is only when they are run together.

I have tested this with MySQL as well as the in-memory SQLite3 database and this doesn't have much of an impact on the speed.

As I mentioned, I am using the new RefreshDatabase trait, which as I understand essentially does the same thing as the DatabaseMigrations and DatabaseTransactions did previously (I know they are still available).

Having looked at the RefreshDatabase trait

protected function usingInMemoryDatabase()
{
    return config('database.connections')[
        config('database.default')
    ]['database'] == ':memory:';
}

Could it be that it doesn't recognise that I am using an in-memory database as I am defining the config values within the phpunit.xml file itself, rather than in the database.php config file. And therefore not doing as I expect?

MikeHopley

Could it be that it doesn't recognise that I am using an in-memory database as I am defining the config values within the phpunit.xml file itself, rather than in the database.php config file. And therefore not doing as I expect?

I think you are correct there. As you can see from the code, Laravel is checking its config files and not inspecting PHPunit's config.

mstnorris

Unfortunately that isn't it as the docs (and my testing) say otherwise. https://laravel.com/docs/5.5/testing#environment

jplhomer

@mstnorris I'm currently experiencing the same issue! Pulling my hair out, and I've tried so many things: sqlite/mysql, withholding certain tests, trying in-memory vs in-database storage. Tests always hang on the ~25th test for me, and the hanging test takes at least 30 seconds to complete. Skipping a test just causes the next one to hang.

............................                                      28 / 28 (100%)

You should really fix these slow tests (>500ms)...
 1. 36563ms to run Tests\Unit\VacationRequestTest:test_a_request_approved_scope_works
 2. 815ms to run Tests\Feature\CreateVacationRequestTest:test_notification_is_sent_when_vacation_requested


Time: 38.85 seconds, Memory: 296.00MB

OK (28 tests, 53 assertions)

I'm running:

Laravel v5.5.21
macOS High Sierra
PHPUnit 6.5.5
PHP 7.1.3

I've noticed that my php shoots to 100% CPU usage, and that the memory used is extremely high when the tests run long.

Experiencing this same issue when running it in a Docker container with PHP 7.2

mstnorris

@jplhomer thanks for your reply. I'm glad I'm not the only one.

I also asked the question on Stack Overflow, if you wouldn't mind leaving a reply there too, giving weight to the fact that it isn't just my setup that would be great.

https://stackoverflow.com/questions/48249205/phpunit-tests-run-using-php7-2-and-7-1-are-3x-slower-than-when-run-using-php7-0

I'll write up some more notes over the coming days as I'd really like to get to the bottom of it.

shez1983

could it be something you are doing in TEstCASE that is run for evey test? that spikes up teh usage/RAM and thus slowing down the test? but then again the stats you showed didnt indicate Laravel/phpunit using more RAM..

mstnorris

@shez1983 I'm using the RefreshDatabase trait so it resets the database after each test.

I can also confirm that running the tests does cause a spike in CPU usage (over 100%), but RAM is not affected. This is however for all three versions (7.0, 7.1, and 7.2) so it doesn't appear to make a difference. PHP7.0 is still approx 3x as fast with the current tests that I have.

dmhamilt

Have you tried turning off broadcasting?

This does not explain everything, but helped me trim off seconds for me.

in .env add/change BROADCAST_DRIVER=log

shez1983

can you turn of refreshDatabase and see what happens then? can you also install valet or test this on a server to see if you get same results there?

Please sign in or create an account to participate in this conversation.