MariaDB THREADS_CONNECTED build up with static php-fpm
Question
Under what circumstance does php-fpm maintain a DB connection beyond the end of a script or request?
The Problem
My app is written in PHP, on top of Laravel 8.
Every hour, I have to restart php-fpm to prevent the MariaDB database from hitting max_connections = 150 and disabling the app since no more connections can be created.
Diagnostic Information
PHP-FPM is configured as static, with a max child count of 39.
DB connections are not configured as persistent.
Raising max_connections above 150 only delays the issue.
There are three DB nodes and three app nodes. The app nodes only talk to their partner DB node in the same region.
The DB nodes are replicating to each other via Galera.
The DB nodes have too many connections independently, not as a cluster.
Checking show full processlist shows me that the vast majority of connections are in SLEEP state and doing nothing.
Using the remote port from the processlist and ss on the app node as well as the php-fpm status page, I've determined that the children holding the connections open are themselves in idle state.
Attempted solutions
I've switched php-fpm to dynamic and set the idle-timout to 10s. The children do not quit, and I can't see any errors.
I've turned down the number of requests a php-child can handle before it is reaped from 100 to 1 with no effect.
I've registered a shutdown handler with PHP that checks if my DB connection is open and produces an alert. No alerts have been sent.
I've set up a cronjob to systemctl restart php7.4-fpm every hour. This alleviates the issue, but obviously isn't a good solution.
Thanks for reading and any idea that might help.
I've also asked this same question on StackOverflow