Pest V4 browser testing - unreliable

I would say that it's generally reliable, the thing is most likely that things sometimes takes a little longer to run/render that you would suspect. It could be a db query, or and API hit that causes the delay, and if something is missing when you try to access it, it might look like random errors.

I've been using Cypress and Playwright to run browser tests, and I had to add extra delays in multiple places to get the tests to green every time.

For each action that require some time to finish, you should always add some pause before doing anything else.

But always remember, the purpose of browser testing or unit testing is that something you would expect to happened.

I setup a script to run tests repeatedly and catch success/failure and I get results like:

Success: 70  Failures: 3752  Total: 3822  Error Rate: 98.168%
Success: 3  Failures: 9  Total: 12  Error Rate: 75.000%
Success: 8  Failures: 8  Total: 16  Error Rate: 50.000%

That big one with the 70 successes I left running for a couple of hours, and when I came back the failure rate was ticking up about 1 count per second and there were a ton of child php processes in ps -ef f, I had to restart the machine.

After a while the mariadb process gets overloaded or something, and sql connections appear to start dropping.

I try to isolate as many variables as possible and to make tests as repeatable as possible, I drop all tables from the test database first, and re-import the same sql dump for each test outside of any pest scripts before running the tests.

Here's what my test script looks like:

$ cat ./test.sh
#!/usr/bin/env bash

success=0
failure=0

while true; do

    if (
        mysql -Nse "
        SET FOREIGN_KEY_CHECKS=0;
        SET @tables = NULL;
        SELECT GROUP_CONCAT(CONCAT('\`', table_name, '\`'))
        INTO @tables
        FROM information_schema.tables
        WHERE table_schema = 'dev_mydb_testing';
        SET @tables = IFNULL(CONCAT('DROP TABLE ', @tables), 'SELECT 1');
        PREPARE stmt FROM @tables;
        EXECUTE stmt;
        DEALLOCATE PREPARE stmt;
        SET FOREIGN_KEY_CHECKS=1;
        " dev_mydbfile_testing \
        && mysql dev_mydb_testing < database/dev_mydb_testing.sql \
        && ./vendor/bin/pest tests/Browser/ProviderActions.php --filter="provider can create first report"
    ) >run.log 2>&1; then
        ((success++))
    else
        ((failure++))
        cp run.log "failure-$(date +%s).log"
    fi

    total=$((success + failure))

    printf "\rSuccess: %d  Failures: %d  Total: %d  Error Rate: %.3f%%" \
        "$success" \
        "$failure" \
        "$total" \
        "$(awk "BEGIN { printf ($failure * 100) / $total }")"

done

By using --filter="provider can create first report" I'm limiting the test runs to just the first test, which modifies the database (hence the need to re-import each time).

Here is the "provider can create first report" test:

test('provider can create first report', function () {

	// load a user
	$user = User::where('email', '[email protected]')->first();// load provider user

	expect($user)->not->toBeNull()
		->and(Hash::check('mypass123', $user->password))->toBeTrue()
		->and($user->email_verified_at)->not->toBeNull();

	$page = visit('/reports')
		->resize(1920, 1080)
		->assertSee(__('Log in'))
		->assertSee(__('Email'))
		->assertSee(__('Password'))
		->assertSee(__('Remember me'))
		->assertSee(__('Forgot your password?'))

		// login
		->type('email', $user->email)
		->type('password', 'mypass123')
		->click('button[type=submit]')
		->assertSee($user->organisation->name)
		->assertSee('does not have any reports.')
		->assertSee('Before you create your first report you need to create at least one Client.')

		// check org
		->click('#btnOrgInfo')
		->assertSee($user->organisation->name)
		->screenshot(filename: '1-check_provider.png')
		->click('#modalUserInfo')

		// do create
		->click('Create Client')
		->assertSee('Sometext')
		->assertSee('Somemoretext')
		->clear('slt')// clear "New client"
		->type('slt', 'alonguniquestring12345')
		->assertSee('alonguniquestring12345')
		->screenshot(filename: '2-before_click_create.png')
		->press('Create')
		->assertSee('alonguniquestring12345')
		->assertSee('created!');

	// create client successful - save screenshot
	$page->screenshot(filename: '3-provider_can_create_first_client_success.png');

	// create first report
	$page = visit('/reports')->resize(1920, 1080);
	$page->assertSee($user->organisation->name.' does not have any reports.')
		->assertSee('Create Report')
		->screenshot(filename: '4-no_reports.png')
		->click('Create Report')
		->type('amount', strval(fake()->randomFloat(2, min: 1, max: 250)))
		->select('select[name="category_id"]', '24')
		->type('description', fake()->text())
		->screenshot(filename: '5-before_create_first_report.png')
		->click('#btnConfirmSave')// just save, do not approve immediately
		->screenshot(filename: '6-confirm_create_first_report.png')
		->click('#btnModalConfirmSave')
		->wait(1)
		->screenshot(filename: '7-report_saved.png')
		->assertSee('Report 78 created')
		->click('#btnCloseReport')
		->wait(1)
		->screenshot(filename: '8-after_create_first_report_DONE.png');
});

Nothing overly complex. But I have tried trimming this back to just the first 1 or 2 screenshots, and I get similar results - it either completes or doesn't complete with about the same frequency.

I'm stumped with this one. I don't think browser tests are going to work for me - I'll probably have to go back to ordinary feature tests and perform the same testing manually before I push updates.