Jobs on :reserved queue but all workers idle
I have a Laravel app that ran fine for many years with Docker Swarm, Postgres and Redis. The queue runs through Redis and the queue workers are managed by Swarm. Recently I'm seeing my queues never falling back to 0 jobs in artisan queue:monitor. It looks like some jobs are pushed to the :reserved queue but no queue worker is working on them. Instead, the workers are idle or continue to process newer jobs (of the same classes).
Has anyone seen something like this? How can a job be moved to :reserved but no worker processes it? In my understanding this can only happen when a worker picks up the job and processes it. These jobs now hang around in :reserved until they are retried based on the retry_after config of the queue.
Please or to participate in this conversation.