Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

mp3man's avatar

Problem with memory size of a giant query over a Collection

Hi, I've a query executed as follows:

$data = Application::select(['id','numinst','nomapp','idapp','fabricante','version','carpeta','comentarios']);
$numTotal = $numRecords = $data->count();

And then I have to get all the results to peform the next operation:


/*
 * Heres is some code that adds optinonally  where clause to the query
*/
if(!empty($searchPhrase)) {



            // Search by a word or part of a phrase (center or user agent)
            if(preg_match("/^[a-z\s]{3,}$/i", $searchPhrase)) {
                $data->where(function($query) use ($searchPhrase) {
                    $query->orWhere('fabricante', 'like', '%' . $searchPhrase . '%');
                       // ->orWhere('osname', 'like', '%' . $searchPhrase . '%');
                });
                $numRecords = $data->count();
                //dd($numRecords);
            }
        }

$totalInstallations = 0;
  
foreach($data->get() as $row) { // TODO: This row takes a lot of memory, #error message: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)
      $totalInstallations += $row->numinst;
 }

The poblem is that the query takes more thatn 128M, due this table contains around 0.5 M rows, and I need to perform the sum of the numinst field, and this is too much memory for my server and very dangerous

I'd like ti know if there's any way to perform the query in laravel as streaming, I mean using the fetch() PDO method, not the equivalent to fetchAll() that uses too much memory. Thanks

Dani

0 likes
8 replies
mp3man's avatar

I don't know why but it worked! I've tested the Eloquent sum() method only, and it didn't take any memory to perform the operation, whereas I use the old foreach it didn't work. Thanks!

jlrdw's avatar

Also if having to iterate over, use the chunk method, see laravel documentation.

mp3man's avatar

Is not a solution, due I want to sum all the values from whole query, not only e single part of it. Your solution would be valid if I split the query in chunks of 100 by 100 or 1000 by 1000 and use the accumulator for each slice of results on each chunk call, but the solution was done by @chaudigv with the Eloquent sum method. Thanks,

Dani

Tray2's avatar

All those operations can and should be done in the database.

Give an example on your data and the expected result and we will be able to help you with that query.

mp3man's avatar

No, I can't do that, due this query is executed with a variable where method, and I cannot make the almost same query two times, one grouping to use the SUM() aggregator in SELECT clause and the other for the normal query to return the data thas is displayed using DataTables in the frontend. Thanks @tray2

Tray2's avatar

I can almost guarantee that running the query twice is still faster and uses way less resources than doing it in php

mp3man's avatar

Hello! I've done it! Now I have the 3 versions of the same code, one that conumes a lot of memory (the foreach over collection), the second that uses Eloquent's method sum() and the last that you proposes: See the beginning of the code, it's the same for 3 versions:

$dataCounter = Application::select(\DB::Raw('SUM(numinst) AS total'));
        //\DB::enableQueryLog();
        $data = Application::select(['id','numinst','nomapp','idapp','fabricante','version','carpeta','comentarios']);

        $numTotal = $numRecords = $data->count();


        //dd($totalInstallations);
        //dd($data);
//dd(\DB::getQueryLog());
        /**
         * Applying search filters over query result as it is was simple query
         */
        if(!empty($searchPhrase)) {



            // Search by a word or part of a phrase (center or user agent)
            if(preg_match("/^[a-z\s]{3,}$/i", $searchPhrase)) {
                $data->where(function($query) use ($searchPhrase) {
                    $query->orWhere('fabricante', 'like', '%' . $searchPhrase . '%');
                       // ->orWhere('osname', 'like', '%' . $searchPhrase . '%');
                });
                $dataCounter->where(function($query) use ($searchPhrase) {
                    $query->orWhere('fabricante', 'like', '%' . $searchPhrase . '%');
                    // ->orWhere('osname', 'like', '%' . $searchPhrase . '%');
                });
                $numRecords = $data->count();
                //dd($numRecords);
            }
        }

In this case, the only problema is that I repeat the same where in $data and $dataCounter builders, that is the only inconvenience I can notice.

Version 1: Foreach (bad way):

foreach($data->get() as $row) { // TODO: This row takes a lot of memory, #error message: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes)
            $totalInstallations += $row->numinst;
        }

Version 2: Eloquent sum method:

$totalInstallations = $data->sum('numinst');

In this case, seems that this call is the same as execute select sum(numinst) due it doesn't consumes resources.

Version 3: Using an alternative query:

$totalInstallations = $dataCounter->first()->total;

Please or to participate in this conversation.