What is the correct way to write a test when paginating the output?

I've been trying to write some tests and if I just use a standard query like; Model::latest()->get() and write a test like so;

        $models = factory(Model::class, 20)
            ->create()
            ->sortByDesc('created_at');

        $this->getJson('/api/models')
            ->assertOk()
            ->assertJson($models->toArray());

It works fine.

However as soon as I implement paginate() on the query instead of get(), the test starts being real funky even after I update the assertJson to be something like; assertJson([ 'data' => $models->toArray() ])

It will pass about 75% of the time and the other 25% it will fail just because the response is in a different order than expected. I can fix it kind of, but it makes my tests incredibly messy and cluttered and I'd rather just not to be honest.

So I'm looking to start fresh. Can anyone suggest me a clean way to assert paginated data in a test? Thanks.