Queue failed_jobs table keeps on populating previous data after updating to PHP 8.1

Hi,

I am experiencing a weird issue related to queues after updating from PHP 7.4 to PHP 8.1. Please read on...

Setup: Laravel 8.83.27 on PHP 8.1 QUEUE_CONNECTION=redis PHP Redis Version: 5.3.7

I have a Redis queue called "emails" for sending system emails. The worker is setup in Supervisor with the following config:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/laravel/artisan queue:work --queue=emails,default --rest=0.5 --sleep=5 --max-jobs=1000 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=root
numprocs=2
redirect_stderr=true
stdout_logfile=/var/www/laravel/storage/logs/worker.log
stopwaitsecs=3600

The emails are delivered correctly via SMTP but somehow the "failed_jobs" table is being populated with rows for exceptions that reference the Mailable class in question. One example error message:

Illuminate\Queue\MaxAttemptsExceededException: App\Mail\ResetPasswordRequest has been attempted too many times or run too long. The job may have previously timed out. in /var/www/laravel/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:750 .......

Once again, the email was delivered pretty much instantaneously.

When I did php artisan queue:flush or manually clear the rows in the table, the same rows came back after about a minute or two.

When I retried a failed job ID using php artisan queue:retry f1bcee70-359b-431a-817e-953d103e3912, the email goes out as usual and then a few moments later the same job ID reappeared in the failed_jobs table.

It is as if there is a disconnect between Redis and Laravel or between the two spawned queue workers if that make sense.

This weird issue was not observed at all before the PHP 7.4 to PHP 8.1 upgrade.

I have ran the usual php artisan cache:clear, queue:restart, queue:clear, config:cache commands in case you are wondering.

What could possibly be the cause here? Please help me to troubleshoot.

Thank you.

LaryAI

3 years ago

Level 58

Well, it looks like you've got a bit of a conundrum here! It sounds like you've done all the right things to try and troubleshoot the issue, so it's time to get creative. Have you tried talking to the queue workers? Maybe they just need a bit of encouragement to get back on track. Or, if that doesn't work, you could try bribing them with some donuts. That usually does the trick!

jlrdw

3 years ago

Level 75

Did you follow every step in the upgrade guide.

shaiful

3 years ago

Best Answer

Level 1

Follow-up updates:

Issue is solved.

It has nothing to do with a faulty PHP 7.4 to PHP 8.1 upgrade. The issue was with my experimentation with Redis compression and serializer in config/database.php by adding these two lines in the Redis options:

'serializer' => Redis::SERIALIZER_IGBINARY,
'compression' => Redis::COMPRESSION_LZ4,

The Igbinary serializer and LZ4 compression are supported as shown in phpinfo() but for some unknown reason it broke the Redis queue regardless if I enabled either one or both. Redis caching and broadcasting didn't seem to be affected though. Since I don't really understand the serializer and compression well enough to be using them, better to just leave them disabled as default I suppose.

Here is a little more info on what's happening with the queue:

I run this on the CLI php artisan queue:work --queue=emails,default --rest=0.5 --sleep=5 --max-jobs=1000 --max-time=3600 and trigger a new job for sending email.
The worker picks up the new job and processes it successfully as normal.
When I run php artisan queue:monitor emails it shows that the queue size is still 1 which means the successful job is not removed from the queue.
A couple of seconds later (maybe a minute), the worker tries to pickup the job again. This time it says "Processing" and then "Failed". This keeps on repeating.

So it appears that the queue fails to clear completed jobs for some reason.

Please or to participate in this conversation.