RefreshDatabase issue with seed data when switching from SQLite to Postgres

I have been advised to change my test db from SQLite to Postgres as I am using Postgres in production (https://laracasts.com/discuss/channels/testing/run-different-migration-in-test?page=1&replyId=796678).

So I started changing the testing environment to Postgres and noticed that RefreshDatabase does not function the same. In my SQLite tests I have the following to seed the database after every refresh and also create users which I can use for my tests using the afterRefreshingDatabase method.

    use RefreshDatabase;
    use SetupAccountsForTestTrait;

    protected $users;


    protected function afterRefreshingDatabase()
    {
        $this->seed();
        // create all the users for the first time
        $this->users = $this->createAllUsers();
    }

When using SQLite the RefreshDatabase option would completely wipe the entire database. However, now with Postgres the database is still intact. As a result if there are more tests in the class, my seeders are seeding again causing issues. Anyway to run the seeds and create the users only once?