ahoi's avatar
Level 5

Laravel Horizon: Define queues per server

Hello there :)

I am trying to solve the following issue:

I got a project that hat different queues: Some short-living tasks and some heavy lifting, which requires a server with somewhat special configuration.

Now of course I am using Laravel Horizon. But I am wondering: Is it possible to define, which server should work on which queues using horizon?

For example I got 3 worker servers: server1, server2 and server3.

I also got two queues:

  1. short-living
  2. heavy-lifting

Now I want server1 to work only on queue short-living. server2 and server3 are supposed to work on queue heavy-lifting, but not on queue short-living.

Of course, all horizon workers would share the same redis backend.

Thank you for your ideas :)

0 likes
1 reply
LaryAI's avatar
Level 58

Absolutely, you can achieve this with Laravel Horizon by customizing the horizon.php configuration file on each server. Horizon allows you to define multiple "environments" or "supervisors" with different queue settings. The key is to have a different supervisor configuration on each server, so each one listens only to the queues you want.

Here's how you can do it:

1. Edit config/horizon.php on each server

On server1 (only process short-living queue):

'defaults' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['short-living'],
        'balance' => 'auto',
        'processes' => 3,
        'tries' => 3,
    ],
],

On server2 and server3 (only process heavy-lifting queue):

'defaults' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => ['heavy-lifting'],
        'balance' => 'auto',
        'processes' => 3,
        'tries' => 3,
    ],
],

You can adjust the number of processes and tries as needed.

2. (Optional) Use Environment-Specific Configuration

If you want to keep a single horizon.php file, you can use environment variables to determine which queues each server should process. For example:

'defaults' => [
    'supervisor-1' => [
        'connection' => 'redis',
        'queue' => explode(',', env('HORIZON_QUEUES', 'default')),
        'balance' => 'auto',
        'processes' => 3,
        'tries' => 3,
    ],
],

Then, set the environment variable on each server:

  • On server1:
    HORIZON_QUEUES=short-living
    
  • On server2 and server3:
    HORIZON_QUEUES=heavy-lifting
    

3. Restart Horizon

After updating the configuration or environment variables, restart Horizon on each server:

php artisan horizon:terminate

Horizon will restart and pick up the new configuration.


Summary:
You can control which queues each server processes by customizing the queue setting in the Horizon supervisor configuration on each server. This is a common pattern for scaling and isolating different types of jobs in Laravel Horizon.

Let me know if you need a more detailed example or help with supervisor process management!

Please or to participate in this conversation.