Troubleshoot slow requests (2% of all requests) found in Nginx log file
Roughly 2% of all requests to our website, take more than 1.4 seconds (average 8.9 seconds). I find these in the access log of Nginx. I use $request_time and $upstream_response_time in log_format to get these numbers.
How would you troubleshoot this further?
One hypothesis could be that the database is slow on these requests, do you agree? How would you analyze that? Check MySQL slow query log? Something else?
I have added a route to the Laravel app. The new route only returns a simple HTML view, nothing with the database, cache etc and even this route can take 10 seconds.
Start by reviewing which routes are slow including query parameters if possible. That can bring you closer already to figure out what might be a problem. It can be a certain page that is slow by doing a filter or search on a database column that does not have an index. Maybe a page that requests something from another service via an API that is slow?
It can be anything, so finding more concrete information on when and where it happens helps you finding the culprit.
@click I believe the routes are random. Sometimes it is the frontpage, other time is a product page etc. It is more like 2% of the time, regardless of route/query something is "stalling" the request in many second.