Horizon queues keep dying

I have a Laravel app setup on an Ubuntu VPS on Google Compute which is uses a Google Memorystore Redis server. For some reason after an indeterminate period of time the Horizon queues 'die' I can't find any explanation for it in the logs for nginx, php, or laravel. If that makes a difference, the server was created via Forge and runs on separate Web and Worker instances. It is not Horizon crashing completely as it is just 1,2, or 3 of the 5 queues that have died.

Neither the worker server nor the redis instance are getting above 20% utilization at any point for both memory and CPU.

Any suggestions about where I can start looking for the rememdy?

LaryAI

2 years ago

Level 58

There could be several reasons why the Horizon queues are dying. Here are a few steps you can take to troubleshoot and resolve the issue:

Check the Horizon logs: Horizon logs any errors or exceptions that occur during queue processing. You can find the logs in the storage/logs directory of your Laravel application. Look for any error messages or stack traces that might indicate the cause of the queue failures.
Monitor the Redis server: Make sure that the Redis server is running properly and is accessible from your Laravel application. You can use the redis-cli command-line tool to connect to the Redis server and check its status. Also, check if there are any errors or warnings in the Redis logs.
Verify the Horizon configuration: Double-check your Horizon configuration to ensure that it is correctly set up. Make sure that the Redis connection details in your config/database.php file match the configuration used by Horizon. Also, check the config/horizon.php file for any misconfigurations or incorrect settings.
Increase the Horizon timeout: By default, Horizon sets a timeout of 60 seconds for each job. If your jobs take longer to process, they might be timing out and causing the queues to die. You can increase the timeout value by modifying the timeout option in your config/horizon.php file.
Monitor server resources: Check the server's resource usage (CPU, memory, disk space) to ensure that it is not running out of resources. Insufficient resources can cause the queues to fail. You can use tools like top or htop to monitor the server's resource usage.

If none of the above steps resolve the issue, you may need to provide more specific information about the error messages or logs you are seeing in order to further diagnose the problem.

// Example of increasing the Horizon timeout in config/horizon.php
'timeout' => 120,

Remember to restart Horizon after making any configuration changes using the horizon:terminate Artisan command.