Have more of a general architecture question. I have the need to upload and import very large files of product data. Could be 100 products or could be hundreds of thousands of products at a single time. I would like have a way to keep the user updated on the progress of the import, but not force them to wait. Would allow a fire and forgot, then check back later on the status type of thing.
I was thinking of writing a command to handle this, then firing that command off when the user uploads the file. The import task itself would write some statuses on progress to a database table (or Redis) which would allow for the status updates.
Just curious how other folks have handled very large file imports. And also to let me know if what I am thinking is not a great idea for reasons that have not occurred to me. Thanks in advance for any feedback. FYI, I am using L5.
Well I haven't ever needed to import really large files, but here are my thoughts.
depending on the types of files you need to upload.. you could use this package as a resource. https://github.com/Maatwebsite/Laravel-Excel I use it to import excel, and csv files to my site, and it is flexible and powerful. but I have no idea if it is viable for hundreds of thousands of products.
I created a FileHandler class that helps me handle my imports... I then created an artisan command line tool that uses the filehandler class so I can import stuff from the command line. I can also use the same filehandler class within the web app as well.
I'm using iron.io to create queues. If you watch the tutorial here on laracasts https://laracasts.com/lessons/ironclad-queues then you'll see its pretty simple to setup and you get alot of power for free. This will allow you to let the user go off and do something and then comeback later and check progress etc. I personally like Iron, because you don't have to set it up yourself, and you don't have to check and make sure it is up and running etc.
I'm working on this very thing. Been a very steep learning curve for me. Using AngularJS $q for client file load into array and FileReader activity. I have the files loading fine and previews (image), working on the progress bar per file on local machine, hope to get that done in the next week. Also, files are uploading fine to Laravel, but the issue I'm struggling with is file upload progress per file from client to server. I think I have to stream as blob to track bytes uploaded.
There are some solutions out there, blueimp, is very comprehensive, but lots of code and a pain to customize, but does have PHP, C and I think Ruby backend code https://blueimp.github.io/jQuery-File-Upload/
I've had to do this with CSVs in the hundreds of thousands to millions. I did actually use the blueimp @nolros mentioned above.
As far as handling the importing once it was uploaded, I used a command, events and queues to handle this. In the background broke the files into smaller more manageable batches.
