iterating over more 5000 cached records in view takes too long

Not sure what you're trying to do or your final outcome for example is this to view is it to print what? If it is to view only you will probably need paginated results. Oh and don't cache a large number of records like that. Furthermore for very large operations it is usually better to write a stored procedure to handle it. Don't do large operations like that inside of your framework or even inside of PHP actually use MySQL. Again unless it's for a users to view and just get results from database and properly paginate.

yes it is to print @jlrdw

@jlrdw the heavy load it not in db or records or caching .. the idea is in iterating over those records in the view which will be printed even in case of expoting themto excel or pdf .. the same situation (the iteration in the view takes BIG time) and that takes long long time

Okay some people won't agree with my solution but it worked 100% dump to local and use a program with an odbc connection setup and print it that way for example at a trucking company I would dump to local I had a odbc connection so I could then link my tables and had some reports setup in Microsoft Access to print. MS ACCESS is what I used but there are other programs you can use to print locally.

Another option would be to do fewer records at a time and Export to PDF. Perhaps span 3 or 4 PDF files. Just a thought.

Edit: In The Dumping to local scenario do the query there just dump the data.

one @jlrdw last question

can i cache the view itself or something like that .. like caching the db results ? so instead of building the view every time i will take the cached one

If this is an internal thing in other words not set up for every user I would set up the report from local then print. The problem with doing it over the web is always resources where as a local database you could literally load 5000 records in a drop-down box for example and that database won't even blink and It can handle it but over the web you would never allow that kind of results in a drop-down box. But again I am just sharing how I have dealt with large reports in the past.

@jlrdw the local solution is not my option now i was just wondering if i can cache the whole view itself

5000 records are quite a few if it was me I would still look at at least exporting the results to PDF or something rather than catching something that large think about it it could grow then what.

@jlrdw this view im taking about is used for exporting to pdf,excel printing so i have to cache this view first to make it faster how i can cache the view ?

Read this http://laravel.io/forum/12-03-2014-fetching-large-numbers-of-rows I'd probably write a raw pdo query, probably more efficient outside of a framework.

@jlrdw ok i found how to render the view

   \Cache::remember('provinces_print',5, function()  {
         return \View::make('province.print')
                ->withProvinces($provinces)
               ->render();
 });

What about paginating the results in groups of 100 or so and then using javascript to fetch the additional pages after the initial page load? You can then simply append the additional info onto the end of the existing info. Im doing this in an app Im making now and its working great.

@zachleigh your idea inspires me and looks awesome but can you give me little steps or any tut or so would be so nice

Are you using a javascript framework?

@zachleigh for frontend no only use jquery

@seco JQuery is Javascript. You use the word cache. Are provinces cached data? If so what type of cache are you using? Laravel views output HTML, are you generating of the HTML? If yes that is the source of your performance slowdown. Reading cache involves overhead based on how the data has been cached.*

Something like this:

• Paginate the results using a custom paginator but don't render the pagination buttons in the view. Instead get the next_page_url property from the paginator instance and put it in a hidden html field or something.
• In your javascript, detect when the page has loaded, get the next_page_url from the html and prefix the url with 'ajax/' or similar so that you can gain control of the pagination. Fire off an ajax call to the url.
• Make a route in your routes file to catch the ajax call. In the controller method, send the request to your custom paginator and return json.
• In your ajax, get the response. Set the hidden html url field to the new next_page_url gained from the response. Process the response data and append it to your html. If the next_page_url is not null, call the ajax method again.

Doing the above would work, but client side processing of the data might be slow. You might want to build the html in some form server side so that you can append it without so much client side work.

i think you should use vue ja and vue resource for that :)

@jeimz173 Thats what I did :) It makes it easier, but its not necessary.

Even rendering a list of 5000 should be pretty snappy. Absolutely make sure you have eager loaded everything and that you don't have n+1 problem. You can do this by monitoring the number of sql queries.

As for creating paged tabular data, I would recommend datatables as this is already geared to working with ajax and will pass a query string to tell your code which 'page' it wants

https://datatables.net/

yes @jimmck im generating views output HTML as you said and i cache it and it is very fast when it is cached of course BUT when some data modification done (update or delete) i clear the cache so it will take more time for the first hit to generate this cached view again so if there is many updates on the table i have to generate the table again and again .. and now that is my problem

@jeimz173 you said i can use Vue and vue resource do you mean that the view can load while im viewing it without waiting the whole view to complete? because i never use Vue before

What's the use case where the user needs to deal with 5000 records?

@Snapey the case is print them all .. 1- i send them to the view (FAST) 2- iterate loop for all of them in the view to generate the html which will be printed (TOO SLOW)

i cache the view so now it is super fast BUT when the data is modified or deleted i have to rebuild that cached view and it takes long time to show this print report nad that is my problem now

What does the controller and the view look like? Have you checked for n+1 issue as recommended?

@seco When you say printing. Do you mean creating a PDF?

@jimmck yes im printing and exporting pdf,excel from the same function because all it does is exporting the generated html for all of them

@seco So you 1) Query the database. 2) Using a Laravel View create HTML representation. 3) Using the HTML output as a source, generate PDF and Excel outputs the user can open in their browser.?

@Snapey the problem was eager loading issue and now it tooks only seconds to show even without caching !!! thanks @Snapey

@Snapey ooops the print goes super fast now but now the excel exporting is still slow although i checked n+1 issue so and the same code used for both printing and exporting

here is the controller code

$province=Province::with('country')->orderBy('name','ASC')->get();

Excel::create($this->reportTitle, function($excel) use ($province) {
                  $excel->sheet($this->reportTitle, function($sheet) use ($province) {
                        $sheet->loadView('province.export')
                          ->withProvinces($province);
                 });
             })->download('xls');

and this is the view code

 @foreach($provinces->chunk(500) as $chunk)
       @foreach($chunk as $province)
         <tr>
                <td>@if($province->country) {{ $province->country->name }} @endif</td>
                <td>{{ $province->code }}</td>
               <td>{{ $province->name }}</td>
            </tr>
        @endforeach
  @endforeach