See my comment on github for a possible solution to the above problem: https://github.com/laravel/framework/issues/39722#issuecomment-975835075
Queue job batching and database overhead of job_batches
I have a high velocity queue with thousands of jobs per minute. I am running Horizon with 4 main queues on 4 main supervisors. Since the process flow requires batching: process ID 123, dispatches a batch of [X, Y] they all dispatch a set of batched jobs X[1,2,3],X[2,3,4]..., Y[1,2,3],Y[2,3,4]..., then when all done, the original ID123 job dispatches a 'finishUpJob' in the Bus' then() branch. Each job can be processed in a sub second, but I am facing an issue with the way Laravel is tracking the batch job progress. Since the only way (as I know of) is the job_batches table in a relational database (using psql right now), I see a lot of "tuple lock" due to the way Laravel is updating the pending and failed jobs fields, pushing the process time for each tiny job to 2-4 seconds due to lock wait just to increment the progress field. And that is a huge issue as the jobs pile up and overwhelm the horizon worker servers and their resources.
Is there any way improve the database writes, or any suggestion how to move the batch jobs tracking to better performing solution, or even skip the batch tracking?
ty, Gabe
Please or to participate in this conversation.