I'm having the same issue now. Did you ever figure this out?
Queued Events "Server reported NOT_FOUND"
Hello,
I've been struggling with an issue for a while, hoping you can help!
It's similar to this issue https://github.com/laravel/framework/issues/15386 but i'm not using pusher or notifications.
I have an event with four handlers, one of the events is a mailer, two ftp files using Flysystem and one makes a Guzzle API call.
It works fine when I the events are fired synchronously, however when I add 'implements ShouldQueue' I get erratic behaviour. Some of the handlers sometimes fire (usually not) and I get the following error:
"exception 'Pheanstalk\Exception\ServerException' with message 'Server reported NOT_FOUND'"
In the failed jobs table I get a max execution error, the worker is set to a 120 timeout and max tries 3.
Thank you in advance for your help! Andrew
I didn't unfortunately, I had to disable queues for certain events. I will be moving to an updated server (PHP 7) on forge soon to see if that helps.
@bryandease @macca This typically happens if your queue timeout is greater than the retry_after or ttr value in queue.php
5.2 - Make sure that the ttr value is greater than the queue timeout value
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'ttr' => 90,
],
5.3 - Make sure that the retry_after value is greater than the queue timeout value
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
],
Have you found any solution for this? I have exactly same issue! It seem to work every now and then, but fails often.
I'm using LV 5.3 and checked that retry_after is value is greater than timeout but it didn't help.
LV version is: 5.3.29 pda/pheanstalk version is 3.1.0
Any ideas?
Some more details:
here is queue.php:
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 300,
]
supervisor conf:
[program:my-laravel-app]
process_name=%(program_name)s_%(process_num)02d
command=php /home/vagrant/Code/MyLVApp/artisan queue:work beanstalkd --sleep=3 --tries=3
autostart=true
autorestart=true
user=vagrant
numprocs=3
redirect_stderr=true
stdout_logfile=/home/vagrant/Code/MyLVApp/storage/logs/worker.log
Illuminate\Queue\MaxAttemptsExceededException: A queued job has been attempted too many times. The job may have previously timed out. in /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:264
Stack trace:
#0 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(197): Illuminate\Queue\Worker->markJobAsFailedIfAlreadyExceedsMaxAttempts('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), 3)
#1 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(152): Illuminate\Queue\Worker->process('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), Object(Illuminate\Queue\WorkerOptions))
#2 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('beanstalkd', 'default', Object(Illuminate\Queue\WorkerOptions))
#3 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('beanstalkd', 'default', Object(Illuminate\Queue\WorkerOptions))
#4 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('beanstalkd', 'default')
#5 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#6 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Container/Container.php(508): call_user_func_array(Array, Array)
#7 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array)
#8 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Command/Command.php(261): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#9 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#10 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(817): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(116): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/vagrant/Code/MyLVApp/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 {main}
@JS I don't see any timeout config in your supervisor configuration file, is there a default timeout for a job? I'm having the same problem as well so I'm trying to look for a solution too.
Ach, sorry. I have tried to set --timeout=240 but it didn't make any difference either. Default value should be 60 if I got it right.
Here is error from logfile:
[2017-01-28 08:29:32] local.ERROR: Pheanstalk\Exception\ServerException: Server reported NOT_FOUND in /home/vagrant/Code/MyLVApp/vendor/pda/pheanstalk/src/YamlResponseParser.php:36
Stack trace:
#0 /home/vagrant/Code/MyLVApp/vendor/pda/pheanstalk/src/Connection.php(142): Pheanstalk\YamlResponseParser->parseResponse('NOT_FOUND', NULL)
#1 /home/vagrant/Code/MyLVApp/vendor/pda/pheanstalk/src/Pheanstalk.php(396): Pheanstalk\Connection->dispatchCommand(Object(Pheanstalk\Command\StatsJobCommand))
#2 /home/vagrant/Code/MyLVApp/vendor/pda/pheanstalk/src/Pheanstalk.php(312): Pheanstalk\Pheanstalk->_dispatch(Object(Pheanstalk\Command\StatsJobCommand))
#3 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Jobs/BeanstalkdJob.php(102): Pheanstalk\Pheanstalk->statsJob(Object(Pheanstalk\Job))
#4 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(285): Illuminate\Queue\Jobs\BeanstalkdJob->attempts()
#5 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(233): Illuminate\Queue\Worker->markJobAsFailedIfHasExceededMaxAttempts('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), 3, Object(Illuminate\Queue\MaxAttemptsExceededException))
#6 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(207): Illuminate\Queue\Worker->handleJobException('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), Object(Illuminate\Queue\WorkerOptions), Object(Illuminate\Queue\MaxAttemptsExceededException))
#7 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(152): Illuminate\Queue\Worker->process('beanstalkd', Object(Illuminate\Queue\Jobs\BeanstalkdJob), Object(Illuminate\Queue\WorkerOptions))
#8 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(75): Illuminate\Queue\Worker->runNextJob('beanstalkd', 'default', Object(Illuminate\Queue\WorkerOptions))
#9 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\Queue\Worker->daemon('beanstalkd', 'default', Object(Illuminate\Queue\WorkerOptions))
#10 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(83): Illuminate\Queue\Console\WorkCommand->runWorker('beanstalkd', 'default')
#11 [internal function]: Illuminate\Queue\Console\WorkCommand->fire()
#12 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Container/Container.php(508): call_user_func_array(Array, Array)
#13 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Console/Command.php(169): Illuminate\Container\Container->call(Array)
#14 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Command/Command.php(261): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Console/Command.php(155): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(817): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(185): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/vagrant/Code/MyLVApp/vendor/symfony/console/Application.php(116): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /home/vagrant/Code/MyLVApp/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(121): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#20 /home/vagrant/Code/MyLVApp/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#21 {main}
```
Hi does someone has an idea how to solve this?
I am on Laravel 5.4 and run exactly into that error message.
Using beanstalkd queue via Forge.
Thanks kay899
I just noticed this same issue today. Here's what I have found so far...it seems to only happen when the failed() method on the Job runs.
I'm still doing some testing, but thought I'd throw this out there in case it helps anyone troubleshoot further.
In my handle() method on the Job, I have some logic that releases the job back into the queue if a condition is not met:
if ($worker->connected)
{
// Some stuff
} else {
\Log::info('SuspendService attempt ' . $this->attempts() . ' failed. Throwing this thing back in the queue');
$this->release(70);
$this->user->notify((new ServiceSuspensionFailed($this->provisioning_record, $this->user))->onQueue('suspensions'));
}
I also have a failed() method in the Job:
/**
* The job failed to process.
*
* @param \Exception $exception
* @return void
*/
public function failed(\Exception $exception)
{
\Log::info('FAILING AND BAILING out on SuspendService after ' . $this->attempts() . ' attempts');
}
If my queue:work command contains only --queue=suspensions --sleep=2
The handle() method will be run over and over again, indefinitely (at least as indefinitely as I was willing to wait before queue:flushing).
If my queue:work command contains --queue=suspensions --sleep=2 --tries=3
This causes the failed() method to be fired on the fourth run of the Job and I see the Pheanstalk\Exception\ServerException: Server reported NOT_FOUND in error.
Like @goatshark I only get this problem when I implement the failed() method. But since I needed a callback for failed jobs I found you can implement a Queue::failing() method inside the AppServiceProvider to do what you need and remove the failed() method inside the job handler. It's worked in my case hope it can help someone else
Ref: https://laravel.com/docs/master/queues#failed-job-events
I've run into this problem as well and what I've found is that I was deleting the job before doing some additional actions (bad idea, I know) and so sometimes one of those actions failed.
I'm assuming the queue driver expects to be able to have the job returned to the queue and then it fails to find it since it was already deleted.
Any Fix for this? Running L5.4 on a forge server timeout is set to 0 and getting the failed jobs exception
Please or to participate in this conversation.