The setup is:
Laravel 5.8 (Upgraded from 5.6)
RabbitMQ as a queue driver (vladimir-yuldashev/laravel-queue-rabbitmq)
Scheduler set up (as described in the docs)
Supervisor configured to run 8 processes of php artisan queue:work --tries=3 --daemon
I have an application running on production & staging. Recently I started to receive a lot of Sentry exception logs.
There are a lot of them, 2 examples:
Core Warning: PHP Startup: Unable to load dynamic library 'xmlrpc.so' (tried: /opt/cpanel/ea-php72/root/usr/lib64/php/modules/xmlrpc.so (/opt/cpanel/ea-php72/root/usr/lib64/php/modules/xmlrpc.so: cannot open shared object file: Too many open files in system), /opt/cpanel/ea-php72/root/usr/lib64/php/modules/xmlrpc.so.so (/opt/cpanel/ea-php72/root/usr/lib64/php/modules/xmlrpc.so.so: cannot open shared object file: Too many open files in system))
Warning: include(/home/www/public_html/vendor/guzzlehttp/guzzle/src/Exception/ConnectException.php): failed to open stream: Too many open files in system
You can notice that there is something in common: Too many open files in system. Other logs also contain this error message, so probably the issue is related to that.
I did some research and found out about checking the ulimit -n, on the server it's equal to 9999
Also, what I have noticed, in the threads related to this issue, they suggest increasing the ulimit to 1024 or 2048. But in my case it's already a lot: 9999
I googled about that too. The max ulimit value can vary, but anyway I feel like there is something wrong in my application, so it produces these errors.
The application has video processing feature: Uploaded videos are converted to mp4 and logo of the company is appended to the left top bottom of the video. All that stuff is done by ffmpeg utility. I am using https://github.com/mikehaertl/php-shellcommand this package to execute shell commands. I am using Laravel queued jobs for video processing, here's the job class: https://gist.github.com/4unkur/e066fd3058b62c1c38808448aa682da7
Also, here are some info I get from debugging:
ps -ef | grep ffmpeg results:
https://pastebin.com/cK09XkLi
lsof:
https://pastebin.com/wMFsmHjX
Any help would be appreciated.
Thanks.