Horizon: Handling priority jobs with supervisor with one process.

In my app, I have a Horizon supervisor set up to run only one process, which works as intended. However, occasionally, there may be a job that needs to be processed with priority.

Would it work to set up multiple queues and dispatch the priority jobs to the high-priority queue? Here’s my current configuration:

'single-process-supervisor' => [
                'connection' => 'redis',
                'queue' => ['high-priority', 'low-priority'],
                'balance' => 'simple',
                'processes' => 1,
                'tries' => 1,
                'maxTime' => 60,
            ],

I’ve seen this approach mentioned online, but I can’t find official documentation about it. Here’s my specific question:

If there are 50 jobs queued in the low-priority queue being processed one by one, and then I dispatch 3 jobs to the high-priority queue, would the worker pick up the jobs from the high-priority queue first, since it's listed first in the array? Or does the order not matter?

Thanks for your help!

LaryAI

1 year ago

Level 58

In Laravel Horizon, when you configure a supervisor with multiple queues, the order in which the queues are listed does matter. The worker will prioritize the queues based on the order they are specified in the configuration array. This means that if you have jobs in both the high-priority and low-priority queues, the worker will first attempt to process jobs from the high-priority queue before moving on to the low-priority queue.

Given your configuration:

'single-process-supervisor' => [
    'connection' => 'redis',
    'queue' => ['high-priority', 'low-priority'],
    'balance' => 'simple',
    'processes' => 1,
    'tries' => 1,
    'maxTime' => 60,
],

If there are 50 jobs in the low-priority queue and you dispatch 3 jobs to the high-priority queue, the worker will indeed pick up the jobs from the high-priority queue first, because it is listed first in the array. Once the high-priority queue is empty, the worker will continue processing jobs from the low-priority queue.

This approach is a common pattern for handling job prioritization in Laravel applications using Horizon. Just ensure that your job dispatching logic correctly assigns jobs to the appropriate queue based on their priority.

vrerabek

1 year ago

Level 3

@LaryAI This is reassuring, but it might be coming from the same source I read online and not from official documentation.

vrerabek

1 year ago

Level 3

Okay just replying to my post if anyone stumbles upon this in the future.

There is, in fact, a mention of this in the official documentation and I totally missed that.

When the balance option is set to false, the default Laravel behavior will be used, wherein queues are processed in the order they are listed in your configuration.

All I have to do is set the balance to false and then it will work as I want it to.

This is my final config of supervisor, which has only one worker prioritizing jobs dispatched into high-priority queue.

'single-process-supervisor-local' => [
                'connection' => 'redis',
                'queue' => ['high-priority', 'low-priority'],
                'balance' => 'false',
                'processes' => 1,
                'maxProcesses' => 1,
                'tries' => 1,
                'maxTime' => 60,
            ],

Tested it and it works!

1 like

Please or to participate in this conversation.