5.1 Factory twice as slow as TestDummy

I've upgraded my application from 5.0 => 5.1 and started swapping out TestDummy for the built in factory "Illuminate\Database\Eloquent\Factory". When I ran my tests it was painfully slow (average was a bit under 2x slower). I really like the additional functionality behind the 5.1 Factory, but is it causing the slowdown?

The results from one of my tests:

Using built in Factory

PHPUnit 4.7.2 by Sebastian Bergmann and contributors.

...

Time: 672 ms, Memory: 26.00Mb

OK (3 tests, 8 assertions)

Using TestDummy

PHPUnit 4.7.2 by Sebastian Bergmann and contributors.

...

Time: 339 ms, Memory: 20.50Mb

Laravel Factory

$factory->define('App\Models\Activity', function($faker)
{
    $date = $faker->dateTimeBetween('-4 months', 'now');

    return [
        'api_id'     => $faker->uuid,
        'user_id'    => 'factory:patient',
        'source'     => 'fitbit',
        'duration'   => $faker->randomFloat(1, 15, 180),
        'distance'   => $faker->numberBetween(1, 15),
        'steps'      => $faker->numberBetween(500, 20000),
        'calories'   => $faker->numberBetween(1000, 5000),
        'vigorous'   => $faker->numberBetween(0, 60),
        'moderate'   => $faker->numberBetween(0, 60),
        'light'      => $faker->numberBetween(0, 60),
        'sedentary'  => $faker->numberBetween(0, 60),
        'date'       => $date,
        'created_at' => $date,
        'updated_at' => $date
    ];
});

TestDummy Factory

$factory('App\Models\Activity', function($faker) {
    $date = $faker->dateTimeBetween('-4 months', 'now');

    return [
        'api_id'     => $faker->uuid,
        'user_id'    => 'factory:patient',
        'source'     => 'fitbit',
        'duration'   => $faker->randomFloat(1, 15, 180),
        'distance'   => $faker->numberBetween(1, 15),
        'steps'      => $faker->numberBetween(500, 20000),
        'calories'   => $faker->numberBetween(1000, 5000),
        'vigorous'   => $faker->numberBetween(0, 60),
        'moderate'   => $faker->numberBetween(0, 60),
        'light'      => $faker->numberBetween(0, 60),
        'sedentary'  => $faker->numberBetween(0, 60),
        'date'       => $date,
        'created_at' => $date,
        'updated_at' => $date
    ];
});