You're overly use one test for many feature, I suggest split it into different test within one test file

• show login page
• user can login
• user can access the specific page
• user can create client
• user can create report
• etc

Thank you for the feedback. Then I think I know this form of testing is not for me, because I will have to have many dozens of highly coupled functions (ie. the order of execution is important because the state of the system is modified by many of the tests).

I setup a script to run tests repeatedly and catch success/failure and I get results like:

Success: 70  Failures: 3752  Total: 3822  Error Rate: 98.168%
Success: 3  Failures: 9  Total: 12  Error Rate: 75.000%
Success: 8  Failures: 8  Total: 16  Error Rate: 50.000%

That big one with the 70 successes I left running for a couple of hours, and when I came back the failure rate was ticking up about 1 count per second and there were a ton of child php processes in ps -ef f, I had to restart the machine.

After a while the mariadb process gets overloaded or something, and sql connections appear to start dropping.

I try to isolate as many variables as possible and to make tests as repeatable as possible, I drop all tables from the test database first, and re-import the same sql dump for each test outside of any pest scripts before running the tests.

Here's what my test script looks like:

$ cat ./test.sh
#!/usr/bin/env bash

success=0
failure=0

while true; do

    if (
        mysql -Nse "
        SET FOREIGN_KEY_CHECKS=0;
        SET @tables = NULL;
        SELECT GROUP_CONCAT(CONCAT('\`', table_name, '\`'))
        INTO @tables
        FROM information_schema.tables
        WHERE table_schema = 'dev_mydb_testing';
        SET @tables = IFNULL(CONCAT('DROP TABLE ', @tables), 'SELECT 1');
        PREPARE stmt FROM @tables;
        EXECUTE stmt;
        DEALLOCATE PREPARE stmt;
        SET FOREIGN_KEY_CHECKS=1;
        " dev_mydbfile_testing \
        && mysql dev_mydb_testing < database/dev_mydb_testing.sql \
        && ./vendor/bin/pest tests/Browser/ProviderActions.php --filter="provider can create first report"
    ) >run.log 2>&1; then
        ((success++))
    else
        ((failure++))
        cp run.log "failure-$(date +%s).log"
    fi

    total=$((success + failure))

    printf "\rSuccess: %d  Failures: %d  Total: %d  Error Rate: %.3f%%" \
        "$success" \
        "$failure" \
        "$total" \
        "$(awk "BEGIN { printf ($failure * 100) / $total }")"

done

By using --filter="provider can create first report" I'm limiting the test runs to just the first test, which modifies the database (hence the need to re-import each time).

Here is the "provider can create first report" test:

test('provider can create first report', function () {

	// load a user
	$user = User::where('email', '[email protected]')->first();// load provider user

	expect($user)->not->toBeNull()
		->and(Hash::check('mypass123', $user->password))->toBeTrue()
		->and($user->email_verified_at)->not->toBeNull();

	$page = visit('/reports')
		->resize(1920, 1080)
		->assertSee(__('Log in'))
		->assertSee(__('Email'))
		->assertSee(__('Password'))
		->assertSee(__('Remember me'))
		->assertSee(__('Forgot your password?'))

		// login
		->type('email', $user->email)
		->type('password', 'mypass123')
		->click('button[type=submit]')
		->assertSee($user->organisation->name)
		->assertSee('does not have any reports.')
		->assertSee('Before you create your first report you need to create at least one Client.')

		// check org
		->click('#btnOrgInfo')
		->assertSee($user->organisation->name)
		->screenshot(filename: '1-check_provider.png')
		->click('#modalUserInfo')

		// do create
		->click('Create Client')
		->assertSee('Sometext')
		->assertSee('Somemoretext')
		->clear('slt')// clear "New client"
		->type('slt', 'alonguniquestring12345')
		->assertSee('alonguniquestring12345')
		->screenshot(filename: '2-before_click_create.png')
		->press('Create')
		->assertSee('alonguniquestring12345')
		->assertSee('created!');

	// create client successful - save screenshot
	$page->screenshot(filename: '3-provider_can_create_first_client_success.png');

	// create first report
	$page = visit('/reports')->resize(1920, 1080);
	$page->assertSee($user->organisation->name.' does not have any reports.')
		->assertSee('Create Report')
		->screenshot(filename: '4-no_reports.png')
		->click('Create Report')
		->type('amount', strval(fake()->randomFloat(2, min: 1, max: 250)))
		->select('select[name="category_id"]', '24')
		->type('description', fake()->text())
		->screenshot(filename: '5-before_create_first_report.png')
		->click('#btnConfirmSave')// just save, do not approve immediately
		->screenshot(filename: '6-confirm_create_first_report.png')
		->click('#btnModalConfirmSave')
		->wait(1)
		->screenshot(filename: '7-report_saved.png')
		->assertSee('Report 78 created')
		->click('#btnCloseReport')
		->wait(1)
		->screenshot(filename: '8-after_create_first_report_DONE.png');
});

Nothing overly complex. But I have tried trimming this back to just the first 1 or 2 screenshots, and I get similar results - it either completes or doesn't complete with about the same frequency.

I'm stumped with this one. I don't think browser tests are going to work for me - I'll probably have to go back to ordinary feature tests and perform the same testing manually before I push updates.

For each action that require some time to finish, you should always add some pause before doing anything else.

But always remember, the purpose of browser testing or unit testing is that something you would expect to happened.

I would say that it's generally reliable, the thing is most likely that things sometimes takes a little longer to run/render that you would suspect. It could be a db query, or and API hit that causes the delay, and if something is missing when you try to access it, it might look like random errors.

I've been using Cypress and Playwright to run browser tests, and I had to add extra delays in multiple places to get the tests to green every time.

Hi all,

Browser testing is awesome, and very powerful. I love being able to script actions and save screenshots.

I'd love to be able to use these, maybe even in some basic CI like github workflows, but for some reason it is just incredibly unreliable for me.

I've worked through everything I can find with online guides and the LLMs etc. I pare down my browser test scripts and test adding one line at a time. I'll get some 20 line script working and then it will randomly start failing, and I'll try reducing it back to a few lines and it's still not working... very frustrating. Things like the screenshots being correct but the script exiting with auth errors, or timeouts etc.

How are people finding browser testing - is it as flaky as my experience, or is it generally reliable?