Level 61
You can split with array_chunk() into 100 record chunks and in each loop report. But 2 million is about 4 times larger than the largest I've done.
Sep 18, 2015
3
Level 12
Server Timeout
I have to import large amounts of data 2million records approx, split into 750k chunks.
I need to compare the import from a CSV against the DB and insert if unique, however this can take over 30 mins.
What's the best approach to do this and report on the progress at the same time?
Does anyone have any experience or can aomeone point me in the right Laravel direction?
Currently it seems that once the process is started in PHP the timer starts also, I thought there might be a way to reset the timer and report on status every 100 records or so.
I've used a commercial site that seems to do this as I see a spinning wheel reload every n seconds in the browser tab ( unless that's just an animated gif :( )
Level 50
I think I'd do this as a commandline artisan task if possible. If not you might want to look into pushing it to a queue and using something like redis to store the current state of progress and querying that. So something like :
Redis::set('import_progress', 0);
while($whatever) {
insertHundredRecords();
Redis::incrby('import_progress', 100);
}
Redis::set('import_progress', 'Complete');
And have a route you can query via JS on the client with something like :
/api/importprogress :
return Redis::get('import_progress');
Level 27
@Luckyfox You could split the process in chunks if the logic of the process permits it, or just dump it on a queue with beanstalkd/supervisord and extend the php execution time a bit.
Please or to participate in this conversation.