artem's avatar
Level 2

Load a lot of data from model into client

Hello all. I have a problem with understanding a logic of a case. I have task of displaying employee list on a page from database with full search and filter possibilities. I have choose vue-tables-2 for this because if i will make search and filter in server side, it can provide a lot of unnecessary database requests. There is simple way load api with axios and pass data into vue component, but problem that i already have more than 50 000+ records and will be more. Maybe some kind of lazy loading can be done here ? Also it's important to have correct search and filters so if loader in some time have only 10 000 records from database and filter applied a lot of data will be skipped and this is really bad. There is no need in Laravel default pagination because vue-tables also have it and it provide search and other possibility. If i will use Cache facade -again how organize the logic? Somehow i need to detect that model was changed and reload cache.

I don't have a lot of experiences especially with laravel, just self studding person with some experiences in freelance, there is no developers community or a lot of good (or known) software companies in my town, just a few who create web sites with WordPress or other CMS and this is my test task for the first office job as developer, so please -any advice will be very helpful...

0 likes
5 replies
DarkRoast's avatar

50,000 is a heck of a lot of records to be grabbing in one query and would make it easy for someone to DOS your site. I think you'd be better off performing those extra database queries.

After you perform operations that modify the database you can just call Cache::forget('key'); or Cache::flush();

1 like
artem's avatar
Level 2

Ok, i think i can use some kind of notifications with events for handle cache flush after modify data. And if i will make search and filter with server side, i do not need vue-table at all. Because there is no need for filtering part of data, just simple sortable plugin can be useful here.

Maybe in this case i can ever use simple pagination and request just 100 records in one time. But do i need cache it on client for data which already received (like return to previous page) or default pagination will be good enough?

artem's avatar
Level 2

And does it take sens to use Cache::remember for about 1 minute for avoiding repeating request with ctrl+f5 ?

DarkRoast's avatar
Level 8

@artem To prevent repeated requests you can use the api middleware which includes a rate limiter - check inside app/Http/Kernel.php.

You could always set something up with ->limit() and ->offset() instead of ->paginate()

1 like
artem's avatar
Level 2

@DarkRoast ok thanks, and what do you think about ::chunk() or maybe ->cursor()? Does it can cause some problems ? Looks like i can just get data by small chunks and put it into cash. Just not sure do i need pass all data to client side if in any case i use search and filter functions on server, or will be better to split data not only for database request but in same way for passing to client.

P.S. and why you think that using of limit() and offset() will be better that simple return Model::paginate() in this case i will receive object and can easy navigate from client to the next page.

Please or to participate in this conversation.