sqlite in-memory doesn't seem to be migrating DB before running tests

Laravel Version 5.2. PHP 7.0.

I have a system that was setup to use MySQL with transactions for the test suite, but I hate that setup and I'm moving it to sqlite in memory.

I have taken the first step of moving to sqlite with the file, and as long as I migrate first and then run the tests everything works fine. So I know the migrations are good.

If I change from the file to :memory: I can still run the migrations manually without any errors, so sqlite still seems happy. There are 52 total migrations. 49 are creating tables, 3 add a single field to a table. But since I can migrate manually I don't think those are the problem.

The problem is any test I run that relies on the DB gets a no such table error.

Migration:

class CreatePhoneNumbersTable extends Migration
{
    public function up()
    {
        Schema::create('phone_numbers', function (Blueprint $table) {
            $table->increments('id');
            $table->integer('type', false, false);
            $table->string('number');
            $table->timestamps();
        });
    }

    public function down()
    {
        Schema::dropIfExists('phone_numbers');
    }
}

Test:

    use DatabaseMigrations;

    /** @test */
    public function a_phone_number_gets_inserted()
    {
        $number = factory(PhoneNumber::class)->create();
        $this->assertNotNull($number);
    }

Factory:

$factory->define(App\Models\PhoneNumber::class, function (Faker\Generator $faker) {
    return [
        'type' => PhoneNumber::EVENT_CONTACT_PHONE,
        'number' => $faker->phoneNumber,
    ];
});

phpunit.xml

<env name="DB_CONNECTION" value="sqlite"/>
        <env name="DB_DATABASE" value=":memory:"/>

error:

Illuminate\Database\QueryException : SQLSTATE[HY000]: General error: 1 no such table: phone_numbers (SQL: insert into "phone_numbers" ("type", "number", "updated_at", "created_at") values (2, 18912874744 x458, 2018-08-06 18:21:53, 2018-08-06 18:21:53))

I have another Laravel site running 5.6, with an almost identical setup and it's working without any problems. I want to get this site upgraded, but I need to write more tests before I'm comfortable doing that.

If I have to just migrate before running tests I can do that, but I really want to just do it in memory with the migrations running.

Thanks for any ideas anyone might have.