Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

Puankare's avatar

laravel loop memory leak

I have a memory leak problem (I think that's what it is called) which I couldn't solve after numerous tries in 24 hours.

Background: I have an articles table (~37M rows) with article_id and journal_id columns, journal_specialty table (~29K rows) with journal_id and specialty_id columns. I want to populate article_specialty table with article_id and specialty_id, using the data from the 2 other tables - writing which specialty(ies) each article is on, based on the journal's specialties they are published on.

My code:

Outcome:

Starting article_specialty population...
Total articles to process: 37765760
        0/37765760 [>---------------------------]   0%
Initial memory usage: 35593704 bytes
Memory usage after fetching articles: 141872128 bytes
Memory usage after inserting: 147389216 bytes
   100000/37765760 [>---------------------------]   0%
Memory usage after processing batch: 41217656 bytes
Memory usage after fetching articles: 145440808 bytes
Memory usage after inserting: 155017720 bytes
   200000/37765760 [>---------------------------]   0%
Memory usage after processing batch: 46857472 bytes
Memory usage after fetching articles: 151319400 bytes
Memory usage after inserting: 161351936 bytes
   300000/37765760 [>---------------------------]   0%
Memory usage after processing batch: 52506008 bytes
Memory usage after fetching articles: 156635200 bytes
Memory usage after inserting: 166937624 bytes
...

It keeps accumulating until it stops with an error, exhausting memory:

PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 1052672 bytes)

What am I missing???

0 likes
5 replies
jlrdw's avatar

toArray(); will eat memory, let the database do the work and paginate.

jlrdw's avatar

@Puankare You aren't trying to load all then paginate are you? You should only be returning from database at most maybe 100 at a time, I normally do 20.

Why so many, just query what's needed.

Puankare's avatar

@jlrdw Sorry, I don’t fully understand your suggestion. I’m working with 37M rows and trying to speed up the process by reading in batches (BATCH_SIZE = 50000) and writing in batches (INSERT_CHUNK_SIZE = 25000). The script works as expected, reading and writing around 2M rows before exhausting memory. After the error, I can rerun it and continue from where it left off. I believe the issue is with memory accumulation between batches and not being able to properly unset accumulated data (what is that is my question), rather than the batch sizes themselves.

Puankare's avatar
Puankare
OP
Best Answer
Level 1

Finally! I got it!!!

The issue was caused by Laravel’s event dispatcher, which was accumulating event listeners during batch inserts, leading to memory exhaustion. The fix was simply to disable the event dispatcher:

DB::connection()->unsetEventDispatcher();

Please or to participate in this conversation.