php5-fpm very high cpu usage

I've seen similar issues over the past several weeks (2 to 3), and they seem to be relating to php being used in conjunction with mysql (either of the process may be stalling), but I have also seen weird I/O Wait spikes in my NewRelic server monitor that seem to occur at the same time. Not quite sure what to make of that.

Update: the IO Wait issue may be due to this: https://docs.newrelic.com/docs/servers/new-relic-servers-linux/troubleshooting/servers-linux-known-issues

@mikebronner Any ideas what I can do to deal with this? Since it's keeping my websites down?

I'm using https://www.digitalocean.com/ as my host provider.

edit: It seems to be very random and I have now idea how I could debug this issue since now php5-fpm is running normal again. http://i.gyazo.com/3d47b584a4186a41cb06c8207491651d.png

@NexPB I have the same problem -- am also on DigitalOcean. I get these spikes in resource usage, that sometimes culminate in the server becoming unresponsive, but not sure what is causing it. The only remedy I have found so far is to reboot the server (since its a production server, we don't have the luxury of keeping our site offline while we do extensive debugging).

In all my years of using Apache/Nginx and php-fpm, I've never had it go above 10%. I run multiple large websites on a single dedicated server. Which VPS are you using from DO?

Well, it appears as mysql is running hot so I would check your queries to see what can be optimized.

I haven't tried it yet, but this app looks promising Blackfire.io

@bashy the 20$/month plan @Kryptonit3 I'll check it out.

EDIT: Even when I run php artisan down php5-fpm will spike from 10% cpu usage to 60-90% all the time.

You would need some serious traffic to make the $20 a month plan worth it.

Your edit: chances are it is a bad query or something similar. Look at your logs to make sure there aren't any errors being thrown by your users that you are unaware of and if you are using google analytics match the times of the spikes with the traffic flow and see what users are doing to cause such issues.

@Kryptonit3 I have 200 users online all the time and at peaks 300, so the 20$ isn't a lot to pay considering my revenue.

@NexPB I'd actually consider getting something larger than that... Even if someone spams your site with a Layer 7 attack, it will reach 60% on fpm workers. If I hold down refresh on one of my sites (forum with a bit of caching) it spikes to 40% on a Xeon E3 quad core.

Definitely check out those 2 links. The debugbar is awesome. And make sure you are leveraging eager loading (N+1 will bog down an app quick).

Running on digital ocean here too - I found the 64bit versions of Ubuntu to be flaky with php5-fpm. Without being too scientific, I swapped over to 32bit and things seem to run better.

@ilovelaravel You crazy? Don't do 32bit. You'll run into a number of issues using that. 64bit php5-fpm is not flaky.

Other possible options are welcome!

@bashy I'm now creating a 80$/month plan droplet and see if it gets any better. If I'm still getting the same issue there has to be something else to it since I never had this issue before.

@NexPB Just spin that up for a few days and see, no need to do a whole month I guess? How many concurrent connections do you get on your sites?

@NexPB I just looked at your site. Nice! From what I can tell, A LOT of the content can be cached. Are you doing any caching?

Also, are you using Nginx or Apache? A site like this would really benefit from an Nginx front-end that proxies requests to apache.

@Kryptonit3 I'm using just Nginx right now, also I'm working right now on trying to optimize everything in the server side while switching to laravel 5.1 from 5.0

@bashy What is the ubuntu command to check the concurrent connections?

To everyone: I change my droplet and it ran really fine again for a few hours and then suddenly it starts spiking again...

Here is my image http://i.gyazo.com/b7cbf2594b7b0d4d3db0ffaa2f42fb4b.png there is also a weird spike in outbound traffic once the cpu start running at 100%, is this simply a ddos?

There's no command for that but you can use something like Google Analytics.

Looks like someone is attacking your site though. 100Mb/s out is 12.5MB/s so 12 people loading a 1MB file every second. Could be a Layer 7 attack (GET/POST).

@bashy I used to have like 300 users at peak hours on my 20$ droplet without issues.

Do you have any tips to try and negate this ddos?

If it's from the same ip you can use the new throttle feature on requests per minute. Setting it high shouldn't affect real users, but not allow repeated requests to execute anything (redirect to 404?). If your stats show average client requests per minute (keeping in mind the number will be inflated if attacks are happening) to just above that number. There are paid services that filter your traffic for you too.

@NexPB It's not a DDoS. You have CloudFlare? You can turn on browser integrity check https://support.cloudflare.com/hc/en-us/articles/200170086

@bashy I already had this turned on before the spike.

If you have a DDoS try lighttpd with mod evasive, had better results with it stopping DDoS attacks than nginx.

Then either add globally server.kbytes-per-second=1024

or per virtual host $HTTP["host"] == "example.com" { server.kbytes-per-second = 64 }

Lighttpd has more traffic shaping settings out of the box. You can also run it as a reverse proxy over nginx so then it would be just a matter of a few minutes to get it up and running.

@bashy it's just my experience, sharing it with others, your mileage may vary! :-)

@ilovelaravel Sure thing but 32-bit OS will force you to limit yourself on a few things like 32-bit integers

@bashy I'm guessing I may have had something in either my nginx config, or php config wrong, but I was running into issues with php5 fpm locking up entirely. The only thing that would bring back the site back would be to reload php5 fpm. This would fix for a completely random timeframe, maybe a day, maybe no more than 10 minutes. After a few weeks of this, I tried a last ditch at 32bit Ubuntu, and it seems to be working at the moment. Maybe just creating a new VPS fixed it, I just don't know for sure.

I concur with the "bad Mysql query" suggestion. I have converted all my "legacy scripts" to artisan commands now, that use Eloquent instead of direct mysqli queries (which were very inefficient the way they were written) and will monitor from here. I have noticed that running console php is much faster than running php-fpm, so I'm hoping it will reduce the load by not having to go through nginx and php-fpm, etc. to run.

Could you maybe post your nginx and php-fpm configs? It is entirely possible that some sort of config quirk is causing your load to spike like that. Unless it really is an attack, then I guess only caching will have a real impact.

DO is a not an ideal VPS host for actual projects. It was a mess when I hosted my sites on it. Linode is a better option in regards of service quality and data recovery. However, it's just from my own understanding.

@lyesqiz What makes you say that? DO and Linode both give you a fully featured virtual machine which allows you to install whatever you want. You can have the exact same system up and running on any of those services. What are the points that where negative about DO for you?

I do have several projects hosted on DO. One customer site for example runs a company blog and easily survived the Reddit effect several times over.

Got the same problem with php5-fpm on nginx and Digitalocean. @NexPB was it a DDos? How did you solved the problem?