Test sometimes passes, sometimes fails

Can you post the actual test code?

@tykus thanks in advance!

This is my Test:

    <?php

namespace Tests\Feature\Models\User;

use Tests\TestCase;
use App\Models\User;
use Illuminate\Foundation\Testing\WithFaker;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Foundation\Testing\DatabaseMigrations;
use Illuminate\Support\Facades\Session;

class ProfileTest extends TestCase
{
    use RefreshDatabase, DatabaseMigrations;

    public function setUp(): void
    {
        parent::setUp();

        // you can call
        $this->artisan('db:seed');

        // or 
        // $this->seed();
    }

    /**
     * Profile See Tests
     */

    /** @test */
    public function a_superadmin_can_see_any_profile_except_other_superadmin()
    {

        $user = factory('App\Models\User')->create();
        $user->assignRole('superadmin');

        $this->signIn($user);

        // a user cannot see profile using /settings/profile/username
        // only using /settings/profile
        $this->get(route('settings.profile.index', $user->username))
            ->assertStatus(404);

        $this->get(route('settings.profile.index'))
            ->assertStatus(200);


        $superadmin = factory(User::class)->create();
        $superadmin->assignRole('superadmin');

        $this->get(route('settings.profile.index', $superadmin->username))
            ->assertStatus(404);


        $admin = factory(User::class)->create();
        $admin->assignRole('admin');

        $this->get(route('settings.profile.index', $admin->username))
            ->assertStatus(200);


        $moderator = factory(User::class)->create();
        $moderator->assignRole('moderator');

        $this->get(route('settings.profile.index', $moderator->username))
            ->assertStatus(200);


        $premium = factory(User::class)->create();
        $premium->assignRole('premium');

        $this->get(route('settings.profile.index', $premium->username))
            ->assertStatus(200);


        $otherUser = factory(User::class)->create();

        $this->get(route('settings.profile.index', $otherUser->username))
            ->assertStatus(200);
    }

This I my TesCase.php:

<?php

namespace Tests;

use Illuminate\Foundation\Testing\TestCase as BaseTestCase;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Illuminate\Foundation\Testing\DatabaseMigrations;

abstract class TestCase extends BaseTestCase
{
    use CreatesApplication;

    protected function signIn($user = null, $password = null)
    {

        $user = $user ?: factory('App\Models\User')->create();

        $user->assignRole('user');

        $this->actingAs($user);

        return $this;
    }
}

I Found My Problem:

The Problem it was that in my UserFactory, I do randomElement in some fields, like:

$factory->define(User::class, function (Faker $faker) {
    return [
        'name' => $faker->name,
        'username' => $faker->unique()->userName,
        'email' => $faker->unique()->safeEmail,
        'api_token' => Str::random(10),
        'isPublic' => $faker->randomElement([true, false])
    ];
});

My testing was depending that a value was always false, but sometimes it was true and other times it was false

Good find @miguellima, I was scratching my head about the likely cause.

Maybe I could suggest that your default factory adopt the happy path (if that means isPublic is true, then so be it). Then use a factory state to set the different value for those special cases:

$factory->define(User::class, function (Faker $faker) {
    return [
        'name' => $faker->name,
        'username' => $faker->unique()->userName,
        'email' => $faker->unique()->safeEmail,
        'api_token' => Str::random(10),
        'isPublic' => true
    ];
});

$factory->state(User::class, 'private', [
	'isPublic' => false
]);

Then it becomes expressive to use it in your tests:

factory(User::class)->states('private')->create();

@tykus Thank you very much for your availability and kindness to help me!

I have fixed it!

note: It was a pain to figure it out! I think more people will stumble in the same mistake!