Performance : From an API to MySQL DB - what's the best way to do this ?

Hi,

I've being trying to figure this out in the last days but with no success.

I have a process that fetches data from an API (each requests takes around 10 seconds to get a response) using queues (with batches and 50 workers running as I can call the API in parallel without getting 429 error code).

When I get a response, I insert the data in the MySQL DB one by one.

I have around 100 calls being made to the API each time the process runs, leading to an average processing time of 10 minutes... I'm looking into shrinking this and getting better CPU usage as it's always giving me 100% CPU.

Does anyone see a better way to handle this ? or a way to really measure what is causing the CPU surge ? DB ? Batch processing ?

I've tried the following :

• writing the data coming from the API to a csv, then importing it to the DB => don't see much difference in overall performance (even though the import from csv to DB is below 3 seconds)

• bulk inserts to the DB by chunking the data from the API but don't see much difference there

Thanks for any idea or if someone has experience with such processes !