Level 88
I sounds like you only have one queue running. Instead you should start multiple queues. I believe you can configure that in amazon or in this case in your worker.
Jul 30, 2018
5
Level 1
Nginx Debug Errors: "Resource temporarily unavailable"
I'm using Laravel 5.4 for my application. There are two tiers; a publically available API and a Worker tier. The worker tier is running on Elastic Beanstalk. The API ships messages to an SQS queue. On the Elastic Beanstalk instances, an Amazon-provided daemon, SQSD, fetches messages from the queue. Each message is forwarded to Nginx as a HTTP request. A specific path is used to process messages from the queue. For the worker tier, here's the full traffic flow:
SQS Queue - SQSD - Nginx - PHP-FPM - Laravel
When I configure nginx to use debug mode for the error log, I'm seeing a huge number of these errors: recv() not ready (11: Resource temporarily unavailable) http upstream recv(): -1 (11: Resource temporarily unavailable)
Nginx is configured to pass traffic to PHP-FPM. We see these errors in our API & Worker stacks.
When these errors were observed, this was the state of PHP-FPM:
pool: www
process manager: dynamic
start time: 30/Jul/2018:15:20:27 +0000
start since: 8834
accepted conn: 27105
listen queue: 0
max listen queue: 0
listen queue len: 0
idle processes: 37
active processes: 1
total processes: 38
max active processes: 36
max children reached: 0
slow requests: 0
Notice that there is 1 active process and 37 processes are idle. At this time, this m5.xlarge instance is using 2.5% of its CPUs. At this time, there are also 80 messages in the SQS queue marked as in flight. It appears we are only capable of processing one message at a time.
When the queue has tens of thousands of messages waiting to be processed, we never see this instance go above 10-20% CPU utilization. As a result, I've had to define my Autoscaling Group to scale up when the CPU utilization is above 10%. I would much prefer to increase the efficiency of this stack and scale up only when the average CPU Utilization is above 90%.
We've removed the session middleware in Laravel because we don't use it and I thought we might be seeing some session locking issues. This didn't make a difference.
Enabling debug mode in PHP-FPM and Laravel hasn't given us any useful information about these "Resource temporarily unavailable" errors. We've also seen the same errors happen in our API. When this happens on the API, I suspect that our users encounter increased latency while Nginx waits for PHP-FPM to accept the request.
My goal is to reduce these errors and increase the efficiency of our worker tier and to decrease the latency of our API tier.
If you need additional information, please let me know!
Level 1
SQS queues are designed for massive throughput. They can support multiple EC2 instances and multiple processes per instance.
Level 88
SQS is only the queue right. Your worker should be the one that is handling multiple processes. So you need to have multiple workers or processes of works running.
Level 1
Yes, SQS is the only queue. php-fpm is configured to have 30 processes / workers.
SQSD runs on the EC2 instance, polls the SQS Queue for messages, and then forwards them to nginx, which forwards them to php-fpm. SQSD is configured to concurrently process 30 messages.
We see the same "Resource temporarily unavailable" message on our API servers, which also run Laravel 5.4. I've set the log level to debug on both laravel and php-fpm, and for the life of me, I can't understand why we're still seeing those errors.
I'll also use markdown to make the php-fpm status readable in the original post. Sorry about that!
Level 1
On our API Tier, these errors mean that something is misconfigured. I'm not sure if it's Nginx, PHP-FPM or Laravel. The end result is that latency is increased for API requests.
On our worker Tier, we've got really shitty throughput for each individual workers, and they rarely process more than 1 message concurrently. Unless I figure out what's going on, the only thing I can think if doing is using much much smaller and cheaper instances, and configure php-fpm for less workers/processes & sqsd for a smaller number of concurrent connections.
Does Laravel 5.4 have any concurrency issues that would be resolved if we upgraded to 5.6?
Can php-fpm and/or Laravel be configured to tell us why it is telling nginx that it isn't ready? Am I encountering some strange session locking bug, even though the Session Middleware has been removed?
Please or to participate in this conversation.