Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

beerbuddha's avatar

Low request per second with Laravel, Docker, and AWS

  • Laravel 5.5
  • Php 7.0
  • AWS 8 cpu / 16GB
  • Docker with nginx/php inside.

I have an api that serves oauth2.0 via passport. It was working fine for low volume traffic but since we have ramped up traffic it buckled. We have a docker application on an EC2 machine with 8 cpu / 16gb ram / SSD. All it the api serves is email/password/client_id/client_secret check, then write the token into the DB, and the user data inside a redis store. (2 read / 2 writes / 1 write to redis). It returns a JWT access_token and a refresh_token (plus the ttl)

The laravel application is dockerized via an ubuntu docker + nginx + 7.0php-fpm with the necessary extensions. (its the same dockerize that Chris Fidao created in his series shippingwithdocker)

I ran locust.io server (on another ec2 machine on AWS) testing at 200 users / 10 hatch/sec and 100% cpu utilization on all cores. If i increase it to 255ish - I start having some 502 bad gateways. I check the php log and php-fpm is unable to service anymore calls. I made a tweak to the php-fpm.conf file: pm = ondemand with 200 children, 1000 max request. I am able to achieve 300 more users at 15 hatch/sec 100% cpu utilization. As another test - what i have done is taken the RAW read DB query + raw insert query and removed redis writting - simply wrote it as part of the routes/api.php closure and let it ripped. I managed to get up to 500 users approx with 134 request/second (no controllers)

My question is: does these metrics seems wrong to anyone? I have an 8 core / 16GB and im getting a low throughput? after finding Taylor Otwell's blog about how he manage to hit 500+ request/sec with a 2GB ram machine on Digital Ocean - is there a setting that I should be checking? I am just trying to ensure that this is my final benchmark metrics and effectively in order to receive more request - I should add another server under a load balancer.

UPDATE #1

  • we use mysql 5.7 (RDS) for the queries and it is 4 queries as i mentioned 2 R /2 W and 1 to redis which is an elasticache instance
  • the median (ms) is 340ms according to locust.io

UPDATE #2 htop screen when under load htop screen

0 likes
30 replies
ohffs's avatar

I think you should be able to handle a lot more traffic than that given the box you have. Does your docker image have xdebug enabled, maybe? Is the DB running on the same box as the application? Do you have php's opcache enabled?

It might be worth looking through the deployment guide and Chris's other video series on scaling with forge might give you some tips (he has a paid one too).

Without seeing exactly what your app does, how it's configured etc - nothing else is coming to mind I'm afraid :-/

beerbuddha's avatar

@OHFFS - hey thanks for the response

  • as i mention all the api is doing is authenticating the user using passport and then writing to redis. I enabled profiling and has determine that pastport / oauth2 does 4 queries - 2 reads and 2 writes. I added the extra redis write due to business objective.
  • the DB is a mysql 5.7 running on a rds instance (so not in the box)
  • I am trying to scale but i am also trying to see (just like a gamer) did i squeeze out all i had on this machine. As you can see (and concluded) this machine SHOULD handle way more
markotitel's avatar

Have you checked RDS metrics? Try the same test with smaller instance and see what you get.

beerbuddha's avatar

@MARKOTITEL - the rds doesn't get tickled. its on low usage and the queries themselves are in the single digits ms. (or 0 << zone). as far as smaller instance goes yes i have. The project used to be on a 2 core 4gb EC2 machine. it choked out at approx less than 80 users at approx 10 request/s

markotitel's avatar

@BEERBUDDHA - Lets see more metrics here.

Do the load test and provide EC2 instance metrics from AWS console.

If possibile install htop or use simple top command wile loadtesting and post screenshots, will be much easier to think of reasons why.

beerbuddha's avatar

@MARKOTITEL - I mentioned it already

  • it taps out all 8 cores. less than 1gb used out of the 16GB.
  • the error message itself is that the php-fpm cannot get service.

from htop It will be included in the main post and here too: htop screen

markotitel's avatar

@BEERBUDDHA - As @ohffs suggested.

Try do add route cache, config cache and optimise autoloader to see what will happen.

Also metrics from AWS Console EC2 instance may hold some info.

beerbuddha's avatar

@MARKOTITEL - I do not own the instance - so i can't tap into the AWS account itself. as far as they are reporting there are nothing choking on that front. I am directly taping into the instance - no load balancer.

  • composer install --no-dev --no-interaction --optimize-autoloader"

already done

ohffs's avatar

As a quick test - I just did a basic benchmark against a little laravel app I have running on a raspberry pi (it uses Redis/Sqlite-on-disk and performs an API lookup to another service on localhost) and I can get 100req/sec with 100 concurrent users - so with your hardware you should be able to handle way, way more than that. It's very odd :-/ Is it possible to try your app on another server outside of a docker container? Some of dockers network modes can have a very big impact on network performance :-/

beerbuddha's avatar

@OHFFS - glad to hear that i am not crazy to think that the app is under performing. I cannot move it away from docker because our infra is built around it and we will eventually move into docker swarm kubernetes system. This is only but one cog. if your follow up question would be: are there any other dockers running on this machine - i would say no not at this moment because we wanted to isolate it in order to run the load test.

ohffs's avatar

I was just meaning to try it on bare metal as a quick test - if that was possible :-) Just to rule out that docker was the issue :-) Docker 'nat' in particular can hit you with a 50% performance overhead :-/ We're moving towards docker swarm/kubernetes too so I've been digging into the networking quite a lot recently :-) I'm not an AWS user so can't offer any advice on that side of things sadly :-/

I don't know if you can try it - but if you have a swarm available - see if you can deploy multiple copies of the app and let something like traefik take care of wiring it all up so requests are split over multiple containers - that's what we're doing just now. But I'm really not sure that would help if you're seeing 100% cpu load across all your cores anyway. In my little raspberry pi test I could only get the 4 ARM cores to max-out at about 80%.

Your app isn't logging a lot of stuff, is it? Or using disk for session/cache? A lot of disk I/O could slow things down a lot too :-/

beerbuddha's avatar

@OHFFS - AWS == (im not triple = it) DO == Linode == any given cloud computing. They are a beast but you are also trap with certain things with them. An EC2 machine is about the same thing that a forge machine would spawn up. So you dont need to be a AWS user - its basically an ubuntu cloud machine just like the rest of them. Just that AWS has better overall eco-system in terms of cost for the volume. shrug not my call.

Doing the docker swarm won't help trying to solve the bottle neck because I dont want to spawn more instances to see how well the load will be. That defeats the purpose of horizontal scaling. Basically its just like putting a load-balancer on top (which i remove to do the test) and having several instances on the bottom. The fundamental difference is that instead of you making the decision you ask the swarm to deal with it as the EC2 instances would be group together.

Now about the logging - yes i do some logging However as i mentioned - If i take out the entire controller idea and just go: "what if i just read 2 queries and write two queries inside a closure routes/api.php file what is my throughput. it fairs better but not best. at 600 users with a 134 req/s. Still no where near where it should

markotitel's avatar

The point was to try this app outside the Docker to see if docker is the bottleneck. But I doubt that.

There is also CPU Credits property on EC2 instances. When depleted your instance will run at 5 - 25% capped.

Try first on some test DigitalOcean, or whatever small VPS to see what you will get there. Setup database on the VPS itself and lets see the results.

beerbuddha's avatar

@OHFFS - Ok so I wasn't able to disable the loging yet however

  • enabled route:cache
  • enabled config:cache

slight improvement (gain a few more users concurrent but I am not getting what you are achieving.

Now opcache - im still trying to figure it out. I tried the opcache one from laravel-news from Paul Redmond but he uses the php-7.2fpm docker container where as i am using the ubuntu one.

i tried using his basic (static) configuration for the opcache.ini and restarted nginx + php-fpm - nothing

as far as the docker is concern it had a opcache.ini and opecache.so zend_extension but that is all i can descern - i couldnt tell if it was active but when i re-ran the load-test i didnt see any improvement.

beerbuddha's avatar

@OHFFS - So i have finally figured out the opcache and it is enabled. opcache enabled confirm settings no improvement unfortunately.

ohffs's avatar

@BEERBUDDHA - I think what @markotitel and I are suggesting is to try it (just as a quick test) on another environment (outside of docker) just to check if docker is contributing to the problem. Not to replace your whole production setup :-) The reason I was suggesting running it outside of AWS was again, just to check another possible reason it could be going slow. AWS is much more complicated animal than, say, Digital Ocean - there are all sorts of things that can be combined that could be having a side-effect.

[Edit - I just saw your new reply] If you add a route to your code that does something like :

Route::get('/whatever', function () {
    phpinfo();
});

then it should tell you if opcache is enabled and what the settings are (there is a section labelled 'Zend OPcache'). [Edit - You probably want to enable the file caching in your opcache setting, try something like :

opcache.validate_timestamps=0
opcache.memory_consumption=128 # MB, adjust to your needs
opcache.interned_strings_buffer=8
opcache.max_accelerated_files=10000 # Adjust to your needs
opcache.max_wasted_percentage=10 # Adjust to your needs

Maybe also start up another container on a different port using the base php image. I usually use php:7.2-apache so I don't have to bother setting up fpm/nginx as well for instance. Just do a laravel new test-project and build it inside the container and see how many requests it can handle. Something like this should do it :

FROM php:7.2-apache

RUN apt-get update && \
    apt-get install -y libfreetype6-dev libjpeg62-turbo-dev libpng-dev libgmp-dev libldap2-dev netcat sqlite3 libsqlite3-dev && \
    docker-php-ext-configure gd --with-freetype-dir=/usr/include/ --with-jpeg-dir=/usr/include/ && \
    docker-php-ext-install gd pdo pdo_mysql pdo_sqlite zip gmp bcmath pcntl sysvmsg exif \
    && a2enmod rewrite

COPY vhost.conf /etc/apache2/sites-available/000-default.conf

EXPOSE 80

COPY . /var/www/html

RUN chown -R www-data:www-data /var/www/html/storage
RUN chown -R www-data:www-data /var/www/html/bootstrap/cache

CMD ["apache2-foreground"]
// vhost.conf
<VirtualHost *:80>
    DocumentRoot /var/www/html/public

    <Directory "/var/www/html/public">
        AllowOverride all
        Require all granted
    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
beerbuddha's avatar

@OHFFS - I think we were responding at the same time. See my response above as I figured out the zend opache and it was enabled.

beerbuddha's avatar

@OHFFS - readjusted to the opcache you have suggested still tapping out at 300 users hatching at 10 users/sec with a 20 req/s

ugh.

ohffs's avatar

If you really can't bring up a test copy for whatever reason, can you install something like php-spx - it might help you narrow down where the time is being spent - but it'll only show you it for the php part - if the problem is elsewhere (docker/aws/nginx/whatever) you'll still be in the dark :-/

ohffs's avatar

You could also try using something like blackfire - there's a free trial which might be enough to get you narrowing down the problem.

beerbuddha's avatar

@OHFFS - thanks for the blackfire suggestion - now i need to figure out how to install it within docker as an agent.

kreitje's avatar

Was your instance created with the PV hypervisor or HVM? We have noticed some issues with PV and disk i/o as well as Redis with PV acting up (SLOOOOWWWWWW).

It's not every client we ran into a problem with, but more of our higher usage clients.

We don't use Laravel so for us, the issue was more sysadmin than our framework or code.

1 like

Please or to participate in this conversation.