It seems like you're experiencing a memory leak while processing a large number of items in a loop. This can happen in PHP scripts that run for a long time and deal with a lot of data. Here are a few suggestions to help you manage memory usage:
-
Disable Eloquent Events: If you don't need model events, you can disable them temporarily which can save memory.
-
Use
unset: After each iteration, unset variables that are no longer needed. -
PHP's Garbage Collection: Force garbage collection after a certain number of iterations.
-
Chunking: Instead of reading the whole file, read it in chunks.
-
Optimize JSON decoding: If your JSON structure is simple, consider writing a custom parser.
-
Use Database Transactions: If applicable, use transactions to reduce the overhead of database operations.
Here's an example of how you might modify your code to incorporate some of these suggestions:
use Illuminate\Support\Facades\DB;
$path = Storage::path("data");
$file = fopen($path, "r");
DB::beginTransaction();
$counter = 0;
$batchSize = 1000; // Adjust the batch size as needed
while ($line = fgets($file)) {
$line = json_decode(trim(trim($line), ","), true);
if ($line) {
Item::create([
'a' => $line['b'],
'c' => $line['d'],
'e' => $line['f'],
]);
}
$counter++;
if ($counter % $batchSize === 0) {
DB::commit();
DB::beginTransaction();
gc_collect_cycles(); // Force garbage collection
}
unset($line); // Unset the variable to free up memory
}
DB::commit(); // Commit the remaining transactions
fclose($file);
In this example, I've added a transaction block that commits every 1000 inserts. This can help reduce memory usage by clearing the query log that Laravel maintains for each transaction. Additionally, I've added a call to gc_collect_cycles() to force garbage collection after each batch.
Remember to adjust the $batchSize to a number that works well with your environment. A larger batch size may be faster but use more memory, while a smaller batch size may be slower but use less memory.
If you're still experiencing memory issues, you may need to look into optimizing your server configuration or using a queue system to process the items in smaller batches over time.
