PHP-FPM is using 100% cpu

Can you ```tail -f```` the log file and see what requests are coming in?

@ohffs Which log file specifically?

@Crinsane Either your laravel log file (try from the base of your laravel install tail -f storage/logs/laravel.log). For the webserver itself do tail -f /var/log/nginx/*. I think you'll have to turn on the access log for nginx though to see much if there aren't errors (at least in homestead it's disabled). So you'd have to do something like sudo sed -i -e's/access_log/#access_log/' /etc/nginx/sites-available/your_site then sudo /etc/init.d/nginx restart then do the tail -f again.

@ohffs there are multiple sites hosted on the server, and indeed on forge the access log is disabled. Also only one of them is a Laravel site, other ones are Wordpress. It's take me a minute to update all the config files to enable access logs. Anything more I can try?

When I stop nginx service nginx stop and I restart php-fpm service php5-fpm restart the processes don't come up anymore. As soon as I restart nginx again, they show up. Does that mean anything to you?

Once you've got the access logs you should be able to see what traffic is coming in. It sounds like something is hitting you a lot. If you see the access logs going crazy with hits then you'll hopefully get an IP address which you can then firewall off at least (for instance sudo iptables -I INPUT -s 123.101.98.135 -j DROP . Could be spammers trying to hit wordpress - I've seen that before sadly :-/

@Crinsane do you mean the fpm process only show up if nginx is running? I don't think you need to restart the fpm process when you restart nginx. Are you able to access your sites ok once they come up? ( I usually use apache with mod_php as I'm not one of the cool kids ;-)

@ohffs what I was trying to say was that disabling nginx seems to stop the php-fpm processes taking up all the cpu. When everything is running, the 5 php-fpm processes take up 20% cpu each. When I stop nginx, the php-fpm processes stop using the cpu. So I guess it's indeed something related to getting a lot of requests or something.

I'm not able to access the sites at all, because when I start nginx, the php-fpm processes start instantly taking up 100% cpu and all I get are 502 or 504 errors when I try to access the sites.

@Crinsane nginx is forwarding the requests to the fpm processes - so that'll be why :-) Have you managed to get the access log up?

@ohffs thanks for the help. I was able to narrow it down to one website and enabled the access log. It showed me a few ip addresses which i blocked using iptables. Now everything seems to be working as expected. I'll keep monitoring it for now, but thanks again for the help!

@Crinsane no problem - glad you're sorted! I do wonder why they disabled the access log in the default config though - seems like an odd decision, but there we go! :-)

@Crinsane - I've been struggling with a similar issue for months now, just stumbled on this thread randomly while looking for something else. Has this continued to solve the problem for you?

What kind of entries did you find in the access log that led you to know what IP addresses to block?

@ohffs - Does the iptables command you specified above (iptables -I INPUT -s 123.101.98.135 -j DROP) simply add an iptables rule to block that IP address from accessing the server entirely? Anything else I would need to know if that is what I would want to do? Are rules added like that persistent, even after a restart?

@intrepidws Forge uses UFW, which is a wrapper around iptables. Here's I suggest:

1. Use UFW over IPtables to make that rule - sudo ufw deny from 207.46.232.182
2. IPtables rules don't persist through reboots by default, however UFW-added rules will (hence the suggestion to use ufw commands over iptables comands directly)

Lastly - yep, any traffic from that IP address will be blocked entirely. You optionally can define a protocol (tcp / udp) or a specific port (80, 443, 22) as well.

@Crinsane see above comment as well, since forge servers use ufw, you may want to add that rule using ufw to be more sure that rule will persist through a server reboot.

@fideloper Wonderful, thanks for that. Any suggestions on how to best identify what IPs I should be blocking, when looking at my access logs?

For anyone interested, I found a good way to search your access logs for IP addresses that appear the most.

sudo cat /path/to/your/access.log |awk '{print $1}' |sort |uniq -c |sort -n |tail

@intrepidws can you explain a bit how the command work?

is it possible to filter down by date and time?

@shiroamada unfortunately that's beyond my knowledge. The command basically takes your access log (/path/to/your/access.log) and then groups it by IP address. It then shows you the count for each IP address.

In my case, it turned out to be caused by the session garbage collector of Laravel. It took a while to find it, but it turns out clearing the storage/framework/sessions folder fixed it. After some research, this was very clearly explained by https://tideways.com/profiler/blog/php-session-garbage-collection-the-unknown-performance-bottleneck