I would move over to MySQL or MariaDB as the test database.
I'm currently running 339 test with closer to 2000 assertions in just over 10 seconds,
Tests: 339 passed
Time: 10.45s
To make that happen I'm using the FastDatabaseRefresh package that you can find here.
https://github.com/PlannrCrm/laravel-fast-refresh-database
I highly recommend it, and I also recommend running your tests against the database server that you will use in production.