Level 88
I see that you run against us-east-2 whenever this error appears. Do you have the correct IAM policy set up that can also run against this region? Without it, these errors might happen.
Otherwise, I have no clue.
Jan 6, 2020
7
Level 2
ChangeMessageVisibility - Intermittent Access Denied
Hey everyone! Happy new year!
Thanks in advance for any help you guys can provide.
I have an SQS queue that appears to work normally most of the time but I keep getting these errors seemingly randomly:
[2020-01-06 22:06:33] qa.ERROR: Error executing "ChangeMessageVisibility" on "https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here"; AWS HTTP error: Client error: `POST https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here` resulted in a `403 Forbidden` response:
<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>A (truncated...)
AccessDenied (client): Access to the resource https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here is denied. - <?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>AccessDenied</Code><Message>Access to the resource https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here is denied.</Message><Detail/></Error><RequestId>d3ab4d17-0848-5c53-9040-1b9ab9e1e40b</RequestId></ErrorResponse> {"exception":"[object] (Aws\Sqs\Exception\SqsException(code: 0): Error executing \"ChangeMessageVisibility\" on \"https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here\"; AWS HTTP error: Client error: `POST https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here` resulted in a `403 Forbidden` response:
<?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>A (truncated...)
AccessDenied (client): Access to the resource https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here is denied. - <?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>AccessDenied</Code><Message>Access to the resource https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here is denied.</Message><Detail/></Error><RequestId>d3ab4d17-0848-5c53-9040-1b9ab9e1e40b</RequestId></ErrorResponse> at /home/app-name/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195, GuzzleHttp\Exception\ClientException(code: 403): Client error: `POST https://sqs.us-east-2.amazonaws.com/123456/sqs-queue-name-here` resulted in a `403 Forbidden` response:
<?xml version=\"1.0\"?><ErrorResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><Error><Type>Sender</Type><Code>A (truncated...)
at /home/app-name/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)
[stacktrace]
#0 /home/app-name/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php(100): Aws\WrappedHttpHandler->parseError(Array, Object(GuzzleHttp\Psr7\Request), Object(Aws\Command), Array)
#1 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(203): Aws\WrappedHttpHandler->Aws\{closure}(Array)
#2 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(174): GuzzleHttp\Promise\Promise::callHandler(2, Array, Array)
#3 /home/app-name/vendor/guzzlehttp/promises/src/RejectedPromise.php(40): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}(Array)
#4 /home/app-name/vendor/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}()
#5 /home/app-name/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(98): GuzzleHttp\Promise\TaskQueue->run()
#6 /home/app-name/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(125): GuzzleHttp\Handler\CurlMultiHandler->tick()
#7 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#8 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#9 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#13 /home/app-name/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#14 /home/app-name/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(58): GuzzleHttp\Promise\Promise->wait()
#15 /home/app-name/vendor/aws/aws-sdk-php/src/AwsClientTrait.php(86): Aws\AwsClient->execute(Object(Aws\Command))
#16 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Jobs/SqsJob.php(57): Aws\AwsClient->__call('changeMessageVi...', Array)
#17 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(383): Illuminate\Queue\Jobs\SqsJob->release('0')
#18 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(341): Illuminate\Queue\Worker->handleJobException('sqs', Object(Illuminate\Queue\Jobs\SqsJob), Object(Illuminate\Queue\WorkerOptions), Object(ErrorException))
#19 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(283): Illuminate\Queue\Worker->process('sqs', Object(Illuminate\Queue\Jobs\SqsJob), Object(Illuminate\Queue\WorkerOptions))
#20 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(118): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\SqsJob), 'sqs', Object(Illuminate\Queue\WorkerOptions))
#21 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\Queue\Worker->daemon('sqs', 'app-name-qa-high,...', Object(Illuminate\Queue\WorkerOptions))
#22 /home/app-name/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\Queue\Console\WorkCommand->runWorker('sqs', 'app-name-qa-high,...')
#23 [internal function]: Illuminate\Queue\Console\WorkCommand->handle()
#24 /home/app-name/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#25 /home/app-name/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#26 /home/app-name/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#27 /home/app-name/vendor/laravel/framework/src/Illuminate/Container/Container.php(576): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#28 /home/app-name/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\Container\Container->call(Array)
#29 /home/app-name/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#30 /home/app-name/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#31 /home/app-name/vendor/symfony/console/Application.php(921): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#32 /home/app-name/vendor/symfony/console/Application.php(273): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#33 /home/app-name/vendor/symfony/console/Application.php(149): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#34 /home/app-name/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#35 /home/app-name/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(133): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#36 /home/app-name/artisan(37): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#37 {main}
"}
I googled around and I have not seen anything that led me to something useful. Any ideas?
Thanks!
Level 2
Yes, everything is running out of us-east-2 and it works most of the time.
I think I have figured it out though. I am still getting it setup but I think this is due to this instance using flat file caches rather than redis. I think the queue workers my be stepping on each other and then the last worker goes to change the message visibility the earlier worker has already taken that action so there is no message to adjust.
Just my current theory. I will report back if that works for others if they run into this issue.
Level 1
@marmour I saw a big uptick in this error starting on 12/28 - I'm still getting them (running in us-east-1). Did you ever figure this out? I'm using db cache (but I don't see why it would matter), but maybe it's time to switch to Redis in any case.
Level 2
Sadly, I have not found root cause. We are using Redis now and we are still seeing a few a day.
It's not clear to me what the issue is as we just put this code into QA so its still pretty green. It's on my list of things to dig into but as of now does not appear to have much of a negative effect aside from a few wasted resources, at least, as far as I can tell.
Level 7
Hey @marmour thanks for reporting that. Would please provide an update now since some time has passed?
Thank you.
Level 2
We ended up switching to Redis queues in part due to this issue.
Would love to know root cause still but I don't have a lot of time to invest in researching it at this point.
Please or to participate in this conversation.