Level 102
Try this
App\Resource::cursor()->map(function($item) {
dump($item);
});
Dec 4, 2019
11
Level 19
Can't get Lazy Collections to work
Hey there! I'm having troubles getting the new Lazy Collections to work.
I'm not working in Laravel, but using the Laravel Container and Database components as composer dependencies directly.
"illuminate/container": "^6.6.1",
"illuminate/database": "^6.6.1",
Everything is working fine so far, I'm using Eloquent Models and the Query Builder as in every other Laravel Application. No problem there.
The problem is there's a huge postgres database with 100.000+ records per table. At one point i want to fetch all instances form a table (the table is called resources) and iterate over each instance, doing some checks, updating, etc... Because of the huge memory load (actually the php process gets terminated after a while) when just doing App\Resource::all()
I want to use Lazy Collections for this task, because as I (thought) I understood it this will resolve that problem altogether.. But something isn't working as I would expect. When doing
$resources = App\Resource::cursor();
dd($resources);
I'll get a Illuminate\Support\LazyCollection back. So far so good.
But when I try to iterate over the results (or even just dump the first result) everything crashes like before, when just doing the App\Resource::all() thing.
What am I doing wrong? Or do I have a misunderstanding on how to use Lazy Collections or what they are for?
Again, to clearify, all these examples crash (php and db process running like wild and get terminated after some time), they're not working for me:
$resources = App\Resource::cursor();
foreach ($resources as $resource) {
// do something
echo $resource->name;
// crashes
}
$resources = App\Resource::cursor();
dd($resources->first()); // also crashes
Would appreciate any help or hints on how to further debug this behaviour..
Thanks a lot in advance!
Level 19
doesn't do anything at all, just goes over it as if the dump($item) isn't even there, and throws an error somewhere in the next lines because the $resources variable is undefined.
that's so weird..
Level 102
I meant, remove the other lines (comment them out) and just do that part alone
App\Resource::cursor()->map(function($item) {
dump($item);
});
die();
Level 19
yeah, I did that. It just doesn't do anything at all.
dd(App\Resource::cursor())
gives me the Illuminate\Support\LazyCollection
everything else, mapping over it and dumping the items doesn't do anything at all.
it seems like there are no items in the collection, but when i'm doing App\Resource::count() it gives me the count of all items...
I'm thinking maybe I do something wrong because just using the "illuminate/database": "^6.6.1", component?
Level 102
Just watched the laracasts video again, and yes it should work with foreach() like your original example :) Sorry
I don't see anything wrong with using that version, no. Laravel just uses self.version meaning it matches the laravel version..
In your original post, you mentioned a "crash". What error are you getting?
Level 19
That's the weird thing. it not really crashes, it's just the php process and postgress process taking up a lot of cpu and memory, and nothing else happens. after a while the server terminates these processes and that's it. So I reckon it's because php tries to fetch 100.000+ records into memory... no error message or anything.
If i do the same thing with less database records everything works fine..
thanks for you looking into my issue! :)
Level 23
what exactly are you doing that you need to fetch 100k records?
Level 102
My thought is that the issue isnt loading the data, but instead the echo'ing of them out to the screen. That would fill the memory I assume.
Perhaps it might be worth just going for you actual use case instead of trying to push them to the screen.
Level 19
It's a table full of html pages. it has all all the html (minified) and some other stuff like the url, and a name stored in there. It's a huge table, the html attribute alone taking up thousands and thousands of characters...
I need to fetch every record, check if the url is still online, update the html if it has changed, parse things within the html, doing some regex, scraping information out of it and this kind of stuff...
the thing is because the table is so huge it's really a pain in the a... and I hoped that lazy collections can help to solve this performance issue...
but i think now that there is some other kind of issue why it's not working.. strange though that everything is working fine when I go with the Resources::all() approach (as long as there aren't to many db records..)
Level 19
@sinnbeck yeah, I'm gonna try this now.. The thing is there is a log file involved with all that, where i should log the results of each iteration... have to take a look into that requirement..
thanks anyway so far! in the first place it was important for me to know that I'm not using the LazyCollections wrong or something.. That seems not to be the case, so I have to continue figuring it out..
Level 1
Had a similar issue where I was looping through hundreds of thousands of records using a lazy collection with grouping and chunking. I was hitting a memory limit, and PHP was silently quitting. After increasing the memory limit of PHP, it continued to run as expected.
For anyone curious, I was using Laravel Herd, so I had to update the PHP memory limit in the UI
Please or to participate in this conversation.