flynnmj's avatar

Pagination issues with large data set.

Laravel 8.x Postgres 12.x

I have an API call with pagination that lets get the status of all their requests. Recently a customer reported an issue with getting all of their results. Inside the controller method is a simple Eloquent query that uses pagination to return the results.

$clientRequests = Request::with(['content', 'user', 'client'])
            ->when($request->start_date, fn ($query, $start_date) => $query->where('add_date', '>=', $start_date))
            ->when($request->end_date, fn ($query, $end_date) => $query->where('add_date', '<=', $end_date))
            ->paginate($perPage)
        ;

The code and pagination is working fine against our test db with about 4,300 records being returned in total. In the production database there are about 15,000 records. Making requests against all the next links, the total number of records returned equals the amount, but we end up with around 9,200 unique records. Switching out the paginate with simplePaginate will up the unique records to around 9,500. Again changing the paginate to cursorPaginate will return the correct number of unique records.

So I think my API calls are going to be converted to use the cursorPaginate. Just trying to determine if I need to be concerned about the pagination of large data set in the blade templates. Has anyone seen similar issues with pagination in Laravel?

0 likes
3 replies
flynnmj's avatar

The primary key is indexed, the add_date column may need to be indexed as well.

Not sure what double pagination would do to improve the duplicate records problem. All relations on this object/table are to one relations.

I am getting the results in the perPage chunk size via the pagination.

flynnmj's avatar
flynnmj
OP
Best Answer
Level 2

In the should have known better category. The Eloquent query needs to have an oderBy() on it so that the database will always return results in a consistent manner.

Please or to participate in this conversation.