Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.
Hi everyone, I import Excel data into a MySQL database with PHP/Laravel. The problem is that it takes time. With a file of 55MB (+500,000 lines) the work is still in progress. Using queues, some tasks fail.Knowing that I want to import more than 10GB of data, how do I optimize it? How to harness the full power of processors? I tell myself when I see the activity of the processors that not all of them are used. NB: I have more than 180GB of Ram and 50-60 CPU. Thank you for the time you will devote to this post.
Hello ... well ... you speak about 180 Gb RAM and 50-60 CPU ... and on the other side a little database (55 Mb / 500 000 lines). I don't understand why you would need so much resources for a little database. I have already imported such files in my database, yes it takes time, and I have only 4 Gb RAM and 2 CPU ;).
Can you be more precise about the performances you need ?
@vincent15000hello, thank you! some files contain 500,000 LINES (55MB), others more than 1million lines (50-60MB. I want to reduce the insertion time.
@cirecodeur I'm not sure that the answer is in Laravel. Perhaps you need some specific material architecture to dispatch the load. Have you thought about AWS ?
@vincent15000 No, Actually, no. I thought if I could distribute the load on different processors it could be done. Multithreaded in php
You can start by using this package, as it should be pretty optimized to work with large files: https://github.com/spatie/simple-excel
You can use multiple processes perhaps (you would need to implement it yourself using spatie/async or similar) but my guess would, that you database would be a bottleneck
@Sinnbeck THANK your answer. I used maatwebsite/excel package. My real problem is to work on reducing import time. I'm supposed to be working more on reducing import time.
Is this a one time import or is it something you need to do on a regular basis?
You can always convert it to csv and install it from the command line . https://phoenixnap.com/kb/import-csv-file-into-mysql
@Tray2 hello, It's something regular for end user. they can upload more than 1GB of Excel(csv, xlx ...) files and the system will import to database. 1 file = 1 table
@cirecodeur Then I would force them to upload CSV and then handle it all with a sheduled cron job, I would remove laravel from the ecuation.
doing it through the framework will always be slower than bulk importing into sql - especially so if you need to check if the record already exists
a single php process will never take advantage of your processors. The amount of ram is probably irrelevant.
some ideas
Please or to participate in this conversation.