Try: php artisan config:cache, if you already haven't.
Deadlock on job's queue even though using redis
Hi
After a lot of fight and research, I didn't succeed to make disappear that error message: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction (SQL: select * from `jobs` where `queue` = default and ((`reserved_at` is null and `available_at` <= 1645604522) or (`reserved_at` <= 1645604432)) order by `id` asc limit 1 for update). This message comes randomly.
Well, firstly I don't understand why I get this message as I don't use yet the queue mecanism: I made all the required settings for later use but currently my code never add jobs to any queue.
Secondly at my wits end because I decided to use a redis server, but the message keeps coming.
I think there is a misconception of my part, and that the server doesn't use at all the redis mecanism, but I can't figure where is the error.
Have someone an idea ?
PS:
- I use laravel 8 (I prefer to wait some time before making the jump), under linux/nginx/mariadb-server-10.5/php7.4
- the queue seems functionnal as:
- I can add jobs :
\App\Jobs\JobNMailTest::dispatch() - I can see the job appearing and disappearing:
Queue::size(); - the job run correctly (as I do receive the mail)
- I can add jobs :
- The queue is managed (started/stopped) with supervisord with something like this:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=/opt/www/html/mywebsite/start_queue.sh /opt/www/html/mywebsite/webroot
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=www-data
numprocs=1
redirect_stderr=true
stdout_logfile=/opt/www/html/mywebsite/worker.log
stopwaitsecs=3600
the script start_queue.sh being:
/usr/bin/php ${ROOT}/root/artisan queue:work --sleep=3 --tries=3 --max-time=3600
- Here is my settings:
-
.env
REDIS_CLIENT=predis QUEUE_CONNECTION=redis REDIS_HOST=127.0.0.1 REDIS_PASSWORD=null REDIS_PORT=6379 -
queue.php
[ 'default' => env('QUEUE_CONNECTION', 'database'), 'connections' => [ 'sync' => [ 'driver' => 'sync', ], 'database' => [ 'driver' => 'database', 'table' => 'jobs', 'queue' => 'default', 'retry_after' => 90, 'after_commit' => false, ], 'redis' => [ 'driver' => 'redis', 'connection' => 'default', 'queue' => env('REDIS_QUEUE', 'default'), 'retry_after' => 90, 'block_for' => null, 'after_commit' => false, ], ], 'failed' => [ 'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'), 'database' => env('DB_CONNECTION', 'mysql'), 'table' => 'failed_jobs', ], ]; -
Here is the error message :
{ "class": "Illuminate\Database\QueryException", "message": "SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction (SQL: select * from `jobs` where `queue` = default and ((`reserved_at` is null and `available_at` <= 1645604522) or (`reserved_at` <= 1645604432)) order by `id` asc limit 1 for update)", "code": 40001, "file": "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:692", "trace": [ "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:652", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:360", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2350", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2338", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2872", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2339", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php:259", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:241", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:219", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:29", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:222", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:345", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:354", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:159", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:117", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:101", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Util.php:40", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Container.php:651", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php:136", "/opt/www/html/mywebsite/root/vendor/symfony/console/Command/Command.php:299", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php:121", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:978", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:295", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:167", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Application.php:92", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129", "/opt/www/html/mywebsite/root/artisan:37" ], "previous": { "class": "PDOException", "message": "SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction", "code": 40001, "file": "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:357", "trace": [ "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:357", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:685", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:652", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:360", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2350", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2338", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2872", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php:2339", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php:259", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:241", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:219", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php:29", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php:222", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:345", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:354", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:159", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:117", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php:101", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Util.php:40", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Container.php:651", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php:136", "/opt/www/html/mywebsite/root/vendor/symfony/console/Command/Command.php:299", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php:121", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:978", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:295", "/opt/www/html/mywebsite/root/vendor/symfony/console/Application.php:167", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Application.php:92", "/opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129", "/opt/www/html/mywebsite/root/artisan:37" ] } } -
the error from root/storage/logs/laravel-2022-02-23.log:
[2022-02-23 10:12:02] production.ERROR: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction (SQL: select * from `jobs` where `queue` = default and ((`reserved_at` is null and `available_at` <= 1645607521) or (`reserved_at` <= 1645607431)) order by `id` asc limit 1 for update) {"exception":"[object] (Illuminate\Database\QueryException(code: 40001): SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction (SQL: select * from `jobs` where `queue` = default and ((`reserved_at` is null and `available_at` <= 1645607521) or (`reserved_at` <= 1645607431)) order by `id` asc limit 1 for update) at /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:692) [stacktrace] #0 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(652): Illuminate\Database\Connection->runQueryCallback() #1 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(360): Illuminate\Database\Connection->run() #2 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2350): Illuminate\Database\Connection->select() #3 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2338): Illuminate\Database\Query\Builder->runSelect() #4 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2872): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}() #5 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2339): Illuminate\Database\Query\Builder->onceWithColumns() #6 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(259): Illuminate\Database\Query\Builder->get() #7 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(241): Illuminate\Database\Query\Builder->first() #8 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(219): Illuminate\Queue\DatabaseQueue->getNextAvailableJob() #9 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php(29): Illuminate\Queue\DatabaseQueue->Illuminate\Queue\{closure}() #10 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(222): Illuminate\Database\Connection->transaction() #11 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(345): Illuminate\Queue\DatabaseQueue->pop() #12 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(354): Illuminate\Queue\Worker->Illuminate\Queue\{closure}() #13 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(159): Illuminate\Queue\Worker->getNextJob() #14 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(117): Illuminate\Queue\Worker->daemon() #15 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\Queue\Console\WorkCommand->runWorker() #16 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle() #17 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #18 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #19 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod() #20 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call() #21 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\Container\Container->call() #22 /opt/www/html/mywebsite/root/vendor/symfony/console/Command/Command.php(299): Illuminate\Console\Command->execute() #23 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run() #24 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(978): Illuminate\Console\Command->run() #25 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(295): Symfony\Component\Console\Application->doRunCommand() #26 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun() #27 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Application.php(92): Symfony\Component\Console\Application->run() #28 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run() #29 /opt/www/html/mywebsite/root/artisan(37): Illuminate\Foundation\Console\Kernel->handle() #30 {main} [previous exception] [object] (PDOException(code: 40001): SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction at /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php:357) [stacktrace] #0 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(357): PDOStatement->execute() #1 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(685): Illuminate\Database\Connection->Illuminate\Database\{closure}() #2 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(652): Illuminate\Database\Connection->runQueryCallback() #3 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Connection.php(360): Illuminate\Database\Connection->run() #4 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2350): Illuminate\Database\Connection->select() #5 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2338): Illuminate\Database\Query\Builder->runSelect() #6 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2872): Illuminate\Database\Query\Builder->Illuminate\Database\Query\{closure}() #7 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Query/Builder.php(2339): Illuminate\Database\Query\Builder->onceWithColumns() #8 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/BuildsQueries.php(259): Illuminate\Database\Query\Builder->get() #9 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(241): Illuminate\Database\Query\Builder->first() #10 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(219): Illuminate\Queue\DatabaseQueue->getNextAvailableJob() #11 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Database/Concerns/ManagesTransactions.php(29): Illuminate\Queue\DatabaseQueue->Illuminate\Queue\{closure}() #12 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/DatabaseQueue.php(222): Illuminate\Database\Connection->transaction() #13 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(345): Illuminate\Queue\DatabaseQueue->pop() #14 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(354): Illuminate\Queue\Worker->Illuminate\Queue\{closure}() #15 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(159): Illuminate\Queue\Worker->getNextJob() #16 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(117): Illuminate\Queue\Worker->daemon() #17 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(101): Illuminate\Queue\Console\WorkCommand->runWorker() #18 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle() #19 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #20 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #21 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod() #22 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call() #23 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\Container\Container->call() #24 /opt/www/html/mywebsite/root/vendor/symfony/console/Command/Command.php(299): Illuminate\Console\Command->execute() #25 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\Component\Console\Command\Command->run() #26 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(978): Illuminate\Console\Command->run() #27 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(295): Symfony\Component\Console\Application->doRunCommand() #28 /opt/www/html/mywebsite/root/vendor/symfony/console/Application.php(167): Symfony\Component\Console\Application->doRun() #29 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Console/Application.php(92): Symfony\Component\Console\Application->run() #30 /opt/www/html/mywebsite/root/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\Console\Application->run() #31 /opt/www/html/mywebsite/root/artisan(37): Illuminate\Foundation\Console\Kernel->handle() #32 {main} "}
-
I think that I have a better understanding: there is one connection default connection[1] and that will be the one used everywhere.
Several points:
-
\App\Jobs\JobNMailTest::dispatch(): you send a job in the default connection and the default queue -
\App\Jobs\JobNMailTest::dispatch()->onQueue('redis'): you send a job in the default connection and a queue named redis -
Queue::size('redis'): you get the number of jobs in the default connection and a queue named redis -
Queue::size(): on get the number of jobs in the default connection and default queue - if
'default' => env('QUEUE_CONNECTION', 'database')is in config/queue.php, then the default connection is the database (unless your .env say otherwise), and you can find the jobs in the database defined as default config/database.php and in the table jobs (there is column for the queue name)
My problems had for cause a mixup between .env/queue.php/dev env/prod env and that the default connection was the dabatase.
Thanks all.
[1] it is said here that you can specify the connection to use, and is said here that you can dispatch a job on specific connection I didn't use those mecanisms
Please or to participate in this conversation.