I finally traced down what was causing this. I'm using dwightwatson/validating which has a Event::until() method. As such, using $this->withoutEvents() breaks.
Overriding the default withoutEvents() function to add until solves the problem. You can add this to your TestCase file or Tests that need to prevent events on watson validating models.
/**
* Mock the event dispatcher so all events are silenced.
*
* @return TestCase
*/
protected function withoutEvents()
{
$mock = Mockery::mock('Illuminate\Contracts\Events\Dispatcher');
$mock->shouldReceive('fire', 'until');
$this->app->instance('events', $mock);
return $this;
}