Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

gaborm's avatar

Queue job batching and database overhead of job_batches

I have a high velocity queue with thousands of jobs per minute. I am running Horizon with 4 main queues on 4 main supervisors. Since the process flow requires batching: process ID 123, dispatches a batch of [X, Y] they all dispatch a set of batched jobs X[1,2,3],X[2,3,4]..., Y[1,2,3],Y[2,3,4]..., then when all done, the original ID123 job dispatches a 'finishUpJob' in the Bus' then() branch. Each job can be processed in a sub second, but I am facing an issue with the way Laravel is tracking the batch job progress. Since the only way (as I know of) is the job_batches table in a relational database (using psql right now), I see a lot of "tuple lock" due to the way Laravel is updating the pending and failed jobs fields, pushing the process time for each tiny job to 2-4 seconds due to lock wait just to increment the progress field. And that is a huge issue as the jobs pile up and overwhelm the horizon worker servers and their resources.

Is there any way improve the database writes, or any suggestion how to move the batch jobs tracking to better performing solution, or even skip the batch tracking?

ty, Gabe

0 likes
1 reply

Please or to participate in this conversation.