Generating a large HTML file without loading entire collection into memory

I need to process a large number of records (50000+) from my DB, pass this data to a view and generate a HTML file (which I convert to a PDF at a later point).

To avoid using too much memory I'm using chunking, selecting the columns that I need and returning the output of a rendered view and that seems to work well.

There's something about my approach when returning the rendered view that doesn't seem right. Almost thinking a cleaner approach is just to write the HTML file without using views.

Has anyone done anything similar to this?


// Controller - Just keep the Builder instance for now. 
$people_on_list = MyModel::select(['col1', 'col2'])->orderBy('col1');


// Parent view
<html>
<body>
<?php
// Call chunk to run a query, getting 10000 chunks at a time
$people_on_list->chunk(10000, function ($people_chunk) {
    echo view('child-view', [
        'people_chunk' => $people_chunk
    ])->render();
});
?>
</body>
</html>

// Child view
loop over $people_chunk creating tables/rows

lostdreamer_nl

8 years ago

Level 53

When you do this in the parent view, you are still caching that entire view until it is done.

What you could do, is use fwrite to stream that output into a new file, and use that HTML file to convert to PDF:

$people_on_list = MyModel::select(['col1', 'col2'])->orderBy('col1');

$file = fopen("test.html", "w");
fwrite($file, '<html><body>');
$people_on_list->chunk(10000, function ($people_chunk) {
    $buf = view('child-view', [
        'people_chunk' => $people_chunk
    ])->render();
    fwrite($file, $buf);
}
fwrite($file, '</body></html>');
fclose($file);

This is still using your childview, but now it will only load 1 childView in memory each time.

When you start converting the HTML to PDF, it will probably still use a lot of memory though.

Please or to participate in this conversation.