Queue Worker Stops Processing Jobs

I have the same problem. When I restart the queues everything works fine. Seems like a problem that many people have.

Serious issue in my production environment. I've had to add some monitors that try to correct the problem and notify me, but I'm not entirely comfortable with that.

Hi Guys. Had the same problem, solved by giving more memory with: --memory=512 (default 128). Seems to work now!

@Deckeron thanks I will try that out.

@deckeron sorry to ask. How do you adjust the memory? Thanks :)

@ardnor you can adjust it in your php.ini on your server. Search for the line memory_limit. If you are using docker, you can find it in your app container /etc/php//php.ini. And in Xamp it's in /xamp/php/php.ini.

Same problem. Sincerely, I don't know what's wrong with laravel queue. They too not good enough to handle the time-consuming task. After a certain time, the queue will stop and will not work fine until you restart them.

And when you ask the question like this even on laracasts, the question will be unanswered. That's the bitter truth bro. Increasing memory, is not the reliable option, what if you have an even bigger task.

I am having the same problems, i restart, everything works, next day nothing works, need to restart again. Weird.

Anyone have ideas? I encounter the same issue, queue always stop working after running for a while, and i have to restart to recover , no more logs for me. To be noticed, i use Redis to store jobs and Laravel Horizon to monitor the queue.

Probably:

php artisan queue:work --queue=your_queue_name --tries=10 --timeout=3000 --memory=8000

--memory is the key.

--memory , didn't solve my problem

I had to reduce the number of processes from 20 to less than 10.

Hello everyone, I'm trying to figure this out. Can you share the exact worker command that you run?

@themsaid I do not have a command but here is Forge setup

connection: database
queue: default
timeout: 30
processes: 2
tries: 3
daemon: yes

Is everyone here using the database driver when this issue happens? Does it happen on different drivers?

@themsaid yes, for me database driver was selected.

Experiencing something similar in the first time. All jobs suddenly stopped (emails are not sent, analytics are not calculated etc). No errors in logs (worker.log, laravel.log), no new rows in failed_jobs table.

My configuration is "1 database worker, 8 redis workers".

For me that was first stop in a few years. When I tried to php artisan queue:restart, only my database queue restarts. Redis queue processes are not killed. I have to kill them manually.

Found a reason: I had buggy job, it has started 8 times, but never ended. All my workers were busy doing basically nothing. So fix is in my code, not in Laravel's. Reading worker.log helped to figure this out.

This also happened to me. I noticed it happened after one job failed for running too long. Didn't see any issue in the logs. Restarting the queue worker didn't help. Only killing it and starting a new one. In the new one I added a 60 seconds limit for each job.

@amosmos maybe you should also add the retry flag so your worker doesnt process a failing job more than x times..

I was also experiencing this same issue, it turns that in one certain process I was performing

 Artisan::call('cache:clear');

which stops my queue everytime, I had to restart the queue manually after that, which was overly annoying for me.

After removing this artisan call everything is working perfectly and queue is not stopping.

One of the best solutions I have found is to create a QueueRestartCommand which will execute every hour or as per your needs and restart the queue

public function handle()
{
    Artisan::call('queue:restart');
    echo "Queue restarted successfully\n";
}

Then add it in your App/Console/Kernel -> schedule()

I also facing the same issue, This happened to me after the Laravel upgrade (7 to 8) After 20 to 26 hours, I have to stop supervisorctl and have to start again(on daily basis), after that jobs are executing properly. NO error logs and no warning, still not able to trace the issue. what I notice is, those jobs which are balance=auto got stops

My Web server (Ubuntu 18.04 Lts) contains php 7.3 Apache and RAM is 78GB, Occupied RAM is around 35GB Maximum so there is no issue of memory leakage.

Here is my Horizon config

@themsaid Please help us out in this regard

<?php

use Illuminate\Support\Str;

return [

    'domain' => null,
    'path' => 'horizonpmdu',
    'use' => 'default',
    'prefix' => env(
        'HORIZON_PREFIX',
        Str::slug(env('APP_NAME', 'laravel'), '_').'_horizon:'
    ),

//    'middleware' => ['web'],
    'middleware' => ['horizoncheck'],
    'waits' => [
        'redis:default' => 60,
    ],

    'trim' => [
        'recent' => 1000, //16.6 Hour
        'pending' => 14400, //3days
        'completed' => 1000, //16.6 Hour
        'recent_failed' => 10080,
        'failed' => 10080,
        'monitored' => 10080,
    ],

    'metrics' => [
        'trim_snapshots' => [
            'job' => 100,
            'queue' => 100,
        ],
    ],

    'fast_termination' => false,

    'memory_limit' => 64,

    'defaults' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default','generate-smsapi-token','logs-citizen-login'],
            'balance' => 'simple',
            'minProcesses'=>1,
            'maxProcesses'=>100,
            'balanceMaxShift'=>10,
            'balanceCooldown'=>3,
            'tries' => 3,
            'nice' => 0,
        ],


        'supervisor-3' => [
            'connection' => 'redis',
            'balance' => 'auto',
            'queue' => ['reset-code-emails','web-password-reset-emails','verification-code-email','pcp-welcome-email','complaint-action-taken-email'],
            'minProcesses'=>1,
            'maxProcesses'=>100,
            'balanceMaxShift'=>10,
            'balanceCooldown'=>3,
            'tries' => 3,
            'nice' => 0,
        ],
        'supervisor-4' => [
            'connection' => 'redis',
            'balance' => 'auto',
            'queue' => ['verification-code-pn','send-android-push-notifications','send-ios-push-notifications','mbalert_incidents','citizen_profile_updated'],
            'minProcesses'=>1,
            'maxProcesses'=>100,
            'balanceMaxShift'=>10,
            'balanceCooldown'=>3,
            'tries' => 1,
            'nice' => 0,
        ],
        'supervisor-6' => [
            'connection' => 'redis',
            'queue' => ['reset-code-sms','pcp-welcome-sms','verification-code-sms','verification-code-mobile'],
            'balance' => 'auto',
            'minProcesses'=>1,
            'maxProcesses'=>100,
            'balanceMaxShift'=>4,
            'balanceCooldown'=>2,
            'tries' => 3,
            'nice' => 0,
        ]


    ],

    'environments' => [
        'production' => [
            'supervisor-1' => [
                'tries' => 3,
                'nice' => 2,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['send-image-faceapp','complaintByDept_level_master'],
                'balance' => 'simple',
                'minProcesses'=>1,
                'maxProcesses'=>50,
                'balanceMaxShift'=>5,
                'balanceCooldown'=>3,
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-3' => [
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-4' => [
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-5' => [
                'connection' => 'redis',
                'queue' => ['new-complaint-created'],
                'balance' => 'simple',
                'minProcesses'=>1,
                'maxProcesses'=>5,
                'balanceMaxShift'=>1,
                'balanceCooldown'=>1,
                //'processes' => 5,
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-6' => [
                'connection' => 'redis',
                'queue' => ['reset-code-sms','pcp-welcome-sms','verification-code-sms','verification-code-mobile'],
                'balance' => 'auto',
                'minProcesses'=>1,
                'maxProcesses'=>100,
                'balanceMaxShift'=>4,
                'balanceCooldown'=>2,
                'nice' => 0,
            ]
        ],

        'local' => [
            'supervisor-1' => [
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['send-image-faceapp','complaintByDept_level_master'],
                'balance' => 'simple',
                'minProcesses'=>1,
                'maxProcesses'=>50,
                'balanceMaxShift'=>5,
                'balanceCooldown'=>3,
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-3' => [
                'connection' => 'redis',
                'queue' => ['reset-code-emails','web-password-reset-emails','verification-code-email','pcp-welcome-email','complaint-action-taken-email'],
                'balance' => 'auto',
                'minProcesses'=>1,
                'maxProcesses'=>100,
                'balanceMaxShift'=>10,
                'balanceCooldown'=>3,
                'tries' => 2,
                'nice' => 2,
            ],
            'supervisor-4' => [
                'connection' => 'redis',
                'queue' => ['verification-code-pn','send-android-push-notifications','send-ios-push-notifications','mbalert_incidents','citizen_profile_updated'],
                'balance' => 'auto',
                'minProcesses'=>1,
                'maxProcesses'=>100,
                'balanceMaxShift'=>10,
                'balanceCooldown'=>3,
                'tries' => 1,
                'nice' => 0,
            ],
            'supervisor-5' => [
                'connection' => 'redis',
                'queue' => ['new-complaint-created'],
                'balance' => 'simple',
                'minProcesses'=>1,
                'maxProcesses'=>5,
                'balanceMaxShift'=>1,
                'balanceCooldown'=>1,
                //'processes' => 5,
                'tries' => 3,
                'nice' => 0,
            ],
            'supervisor-6' => [
                'connection' => 'redis',
                'queue' => ['reset-code-sms','pcp-welcome-sms','verification-code-sms','verification-code-mobile'],
                'balance' => 'auto',
                'minProcesses'=>1,
                'maxProcesses'=>100,
                'balanceMaxShift'=>4,
                'balanceCooldown'=>2,
                'nice' => 0,
            ]
        ],
    ],
];

Following is my Supervisor Config

[program:laravel_horizon]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/lara8/artisan horizon
autostart=true
autorestart=true
redirect_stderr=true
user=www-data
stdout_logfile=/var/www/html/lara8/storage/logs/horizon.log

I had the same issue when on production, I found some jobs in the db. And then I realised that the queues aren't working.

Tried php artisan queue:restart (multiple times) but it didn't fire any queue worker.

After juggling with few things, It tried to clear cache:

php artisan cache:clear

Then,

php artisan queue:restart

Somehow, this command did the trick and queue worker started. Let's see how long they run before stopping again.

Finally, I found the root cause of the problem. Redis has an internal timeout for connections which in Laravel is defined in the queue.php under the retry_after key. We don't see any log because our worker closes immediately after this timeout, so there is no application-level log. Here is my code:

//queue.php
'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver'      => 'database',
            'table'       => 'jobs',
            'queue'       => 'default',
            'retry_after' => 90,
        ],

        'beanstalkd' => [
            'driver'      => 'beanstalkd',
            'host'        => 'localhost',
            'queue'       => 'default',
            'retry_after' => 90,
            'block_for'   => 0,
        ],

        'redis' => [
            'driver'      => 'redis',
            'connection'  => env('REDIS_QUEUE_CONNECTION', 'queue'),
            'queue'       => env('REDIS_QUEUE', 'default'),
            'retry_after' => 1800,
            'block_for'   => null,
        ],

        'redis-long-running' => [
            'driver'      => 'redis',
            'connection'  => env('REDIS_QUEUE_CONNECTION', 'queue'),
            'queue'       => env('REDIS_QUEUE', 'default'),
            'retry_after' => 3600,
            'block_for'   => null,
        ],
    ],

and then in the horizon.php (because I use horizon for managing queues):

    'environments' => [
        'production' => [
            'shortRunningQueue'   => [
                'connection' => 'redis',
                'queue'      => ['shortRunningQueue'],
                'balance'    => 'simple',
                'processes'  => 16,
                'tries'      => 3,
                'nice'       => 0,
                'timeout'    => 1800,
            ],
            'longRunningQueue'   => [
                'connection' => 'redis',
                'queue'      => ['longRunningQueue'],
                'balance'    => 'simple',
                'processes'  => 16,
                'tries'      => 3,
                'nice'       => 0,
                'timeout'    => 3600,
            ],
        ],

        'test' => [
            'shortRunningQueue'   => [
                'connection' => 'redis',
                'queue'      => ['shortRunningQueue'],
                'balance'    => 'simple',
                'processes'  => 16,
                'tries'      => 3,
                'nice'       => 0,
                'timeout'    => 1800,
            ],
            'longRunningQueue'   => [
                'connection' => 'redis',
                'queue'      => ['longRunningQueue'],
                'balance'    => 'simple',
                'processes'  => 16,
                'tries'      => 3,
                'nice'       => 0,
                'timeout'    => 3600,
            ],
        ],
    ],

Now, if I dispatch a job on the shortRunningQueue the job automatically fails after 30 minutes, but if I dispatch it on the longRunningQueue It can run up to 1 hour.

At last, if your application heavily relies on the jobs, I advise you to use a message broker, instead of an in-memory database.

Exact same issue, the job that fail must handle like 5000 lines of data in a Collection, then the execution stops when it reaches like 11 line, sometimes 30.

I did add a timeout (800s) on the job + Guzzle request(2s) + queue config with a retry_after (1200s) bigger than the timeout. But still the job fail with no logs (If I use the handle failing function to print what's wrong it's says timeout).

This not ok since after 5 seconds it's running but then stops for no reason (pretty sure not the code). When I check the logs of supervisord : (terminated by SIGKILL; not expected).

I have 3 workers, is this enough ? should I add more memory ?

Hi, today I face the same problem, the jobs don't fail at fail_jobs table and laravel log doesn't have the execption log.

The supervisor I used is here, build for windows OS, /ochinchina/supervisord

I have ten workers, at first time, I could see that ten jobs was dispatched to worker,

but after it remain less than ten, sometimes 4 or 5.

And I need to restart the supervisor to excute the jobs.

I use laravel 8 and php 8.

Is anyone face the problem before?

optimize:clear worked for me. using Laravel 10