Potti's avatar
Level 1

Handling over 7k records, groupBy and pagination

I have a situation that can't get my head around. In database I have over 7k records. So pagination at server side best thing to do. But I can't do that because I must use groupBy on the data. Official Laravel Docs says "pagination operations that use a groupBy statement cannot be executed efficiently by Laravel." and It's offering a frontend custom pagination.

What is best way to handle this kind of situation. After using groupBy should I just send all 7k to client side and apply custom pagination. Is it right way to do it? Also if I use custom pagination in frontend. Then how can I follow the page links? I mean dynamicly slicing the array and there is no url path =2 or etc... to follow the pages...

0 likes
6 replies
jlrdw's avatar
jlrdw
Best Answer
Level 75

No do not put that many records in an array, see these links

https://laracasts.com/discuss/channels/guides/length-aware-paginator

And a sample report

https://laracasts.com/discuss/channels/guides/getpdo-usage

Use a slice on small data, normal pagination math for a larger amount of data.

Also outside of laravel, I would highly advise you learn how to write pagination and how it all works it's just simple math.

And there's much more detail on the length aware paginator in the laravel API.

2 likes
Snapey's avatar

It depends very much on the nature of the data that you want to paginate (its not just simple math)

So when you group, you end up with parent and children records. Do you want to paginate the parents or the children or both?

Consider if you have 7000 records and when you group them, the first parent has 80 children. Do you want to display all children? Suppose you only display the parent and 50 children. When you click on the next page, how would the paginator know you want the same parent record and the next 30 children, and perhaps some of the next parents?

Do you see the problem?

We can't guess at your use case, what the data is or how it might be grouped, but perhaps an alternative strategy is required?

2 likes
jlrdw's avatar

Well joins with a group by would be single pagination or displaying a top and bottom separate you would need double pagination but still that's easy to do.

You have a top link by parent for next parent and a bottom pagination link for next child.

Irregardless you need to dig in and learn this stuff and work out some complex pagination if your projects are going to have this kind of stuff.

If you've never written your own paginator now is the time to write one just to learn how it all works.

2 likes
jlrdw's avatar

Also a previous answer I gave:

You could write an eloquent join query and paginate, example is not your data but call pet owner the main comment and list of his or her pets nested comments, same idea.

Here jimmie has 3 pages, when you click page 4 it goes to another pet owner.

Sorry just done quick no formatting.

I just did this query as a quick example:

$pets = Powner::query()->leftJoin('dc_pets', 'dc_powners.ownerid', '=', 'dc_pets.ownerid')
                ->select('dc_powners.ownerid', 'dc_powners.oname', 'dc_pets.petname')
                ->orderby('dc_powners.ownerid')
                ->paginate(5);

Page 1, page 2, page 3 is jimmie or say main comment

page 4, page 5 is diane

Under is list of pets, in your case sub (nested) comments.

Or

Just double paginate:

Have a "next main comment" button and a link for next set of nested comments.

This nesting can take a little work, usually no quick easy solution. @Snapey once posted a double paginate solution: http://novate.co.uk/using-multiple-pagination-links-on-one-page/

Or

Use ajax with divisions.

1 like
Tray2's avatar

I would create a view that represents the data set and paginate from that.

CREATE OR REPLACE VIEW some_view AS
SELECT some_field, sum(some_other_field)
FROM some_table
GROUP BY some_field;

Then I can run a simple Eloquent query on it

$result = SomeView::paginate(25);

This example is very simplified.

You can use a migration to create the database view

In your up() method

DB::statement("CREATE OR REPLACE VIEW some_view AS
SELECT some_field, sum(some_other_field)
FROM some_table
GROUP BY some_field");
2 likes
Potti's avatar
Level 1

Thank you for all the answers. In the end I solved the problem by handling the pagination at server side. It's clearly right choice if you have lots of records... As he @jlrdw suggest I used Illuminate\Pagination\LengthAwarePaginator; class.

`

   $places = Place::with(["plans"])->get()->groupBy("place_name")->toArray();
                    
    $total = count($places); 
    $perPage = 11;
    
    $currentPage = LengthAwarePaginator::resolveCurrentPage();

    $currentItems = array_slice($places, $perPage * ($currentPage - 1), $perPage);
    
    $paginator = new LengthAwarePaginator($currentItems, count($places), $perPage, $currentPage);

    $results = $paginator->appends('filter', request('filter'));


    return $results;  

`

3 likes

Please or to participate in this conversation.