Crazy DigitalOcean Response Durations

Good day everyone,

I'm hoping someone out there can tell me what the heck is going on here... Anyone that can point me in the right direction will earn a case of beer from me.

We've deployed a Laravel 9 app to digital ocean for some testing. Here's the setup:

• 1 application droplet with 4 dedicated cores, 8gigs of ram.
• That sits behind a digitalocean-managed load balancer.
• We're using a digitalocean database "cluster" for redis and mysql. At the moment, both clusters only have a single node, both of which are "basic" (which means a shared cpu, from what I gather).
• Redis is a single core, 2gb ram setup while mysql is 2 cores, 4gb ram. According to digital ocean, this redis setup should be good for 10,000 simultaneous connections and 200 new connection per second. Mysql should be good for 225 simultaneous connections. Redis is used for session storage (+ general cache & horizon queues).
• The application droplet has PHP8.1 configured for a php-fpm pool of static and 100 processes.

Ok. Here's the crazy part. (The following stats are partly gathered by hitting our api routes and also from loading a Nova resource page that has a lot of relations. For us, a single page load spawns about 5 initial ajax requests, then a chunk of 20, and then another chunk of 20)

• If the server is handling between 1 and 5 simultaneous requests, it can consistently respond within 200ms, many within 50-150ms. This is roughly in line with my localhost testing (M1max macbook).
• If the server is handling more than 5 requests at once (say, 20), it might respond in 200ms for a few and then all of the others will take between 5100 and 5300ms! Yes, almost exactly 5 seconds longer than a typical request's processing time.
• If I hit the server with even more concurrent requests (100ish), response times will be a few hundred milliseconds, then 5.1-5.3 seconds for a chunk, then 10.1-10.5 seconds for the next chunk, etc.

This probably goes without saying, but all of the resource monitoring stats on digital ocean show very low consumption. I don't see anything that's even close to maxed-out when doing these tests.

I have been struggling to figure out what in the world could be causing this to happen. I've not seen something like this before. Please, for my sanity, someone enlighten me.