I need to send batches of approximately 500,000 messages to an API. The API allows me to send batches of 5,000 messages in each request.
I tried to create a job that builds an array with these 5,000 requests, which are computed through a foreach loop that retrieves information from the database, processes it, and then sends it to a Guzzle Pool. However, when the job takes too long, it is restarted, causing some requests to be sent repeatedly.
I would like your help on what would be the best way to send these requests, whether by increasing the job timeout or by creating several smaller jobs.
Are you saying a single job of building an array of 5,000 elements took too long and restarted (you create many jobs in a loop)? Or did you build 100 of these 5,000-element arrays in the same job, and that’s what made it take too long (you have a loop in a single job)?
I would calculate the start and end rows in the database for each 5,000-element array 100 times in a loop; then for each iteration, send the details to a job queue that calls a job that does the actual array creation and sends the API call.
If each job still takes too long and times out, either increase the timeout period or calculate smaller arrays. If your job is taking more than the default 60 seconds to build an array of 5,000 elements, that must be some seriously heavy data retrieval/processing you’ve got going on, so smaller arrays would probably be wise at any rate in that case.
@kokoshneta thanks for your answer! I think this is where my problem lives. I'm doing everything in one job.
I was thinking after I made the post, if I try to chunk this query and dispatch a job with this chunk, maybe this can work, right? Or create a large amount of jobs will be a problem?
@ricocrivelli Creating a large amount of jobs shouldn’t be a problem – that’s what queues are for. The server will handle the jobs sequentially, as resources become available, while still handling other requests.
Loading the endpoint where you create all the jobs may take a little while since it will be making quite a few database or API calls (to store 100 jobs in the queue), but probably no more than a few seconds all told.
Just make sure you don’t load that endpoint again, ’cause then you’ll end up with duplicate jobs in the queue!
