tr8norman's avatar

[LARAVEL HORIZON] How to specify which Queue Connection Laravel Horizon uses

I have 2 Redis connections (Redis A and Redis B) on my Laravel application.

My Laravel application (consists of multiple servers/queue workers, lets call it Application X) submit jobs to Redis A queue and process the jobs in its workers. Application X uses horizon to monitor the queue and trigger the workers.

Application X also submit jobs to Redis B, but the one processing the jobs submitted to Redis B is a different Laravel application (let's call it Application Y). Application Y also has a Horizon enabled and it also has multiple servers to act as queue workers. Application Y only have 1 Redis connection to Redis B (no connection to Redis A)

My questions is:

When setting up Horizon on server A, I have 2 "driver" defined in under "redis" connection on database.php for Redis A and Redis B. On queue.php, I have also defined 2 connections with "driver" as "Redis". How does Horizon in Application X knows which queue connection to use? Is it simply taking the default queue connection (which is taking value from env('QUEUE_CONNECTION', 'sync')) Or is it simply monitoring all Redis-based queues defined in queue.php? I noticed that I can still see jobs that I submitted to Redis B in Pending jobs of horizon of Application X.

0 likes
7 replies
neilstee's avatar

@tr8norman you can do this using the onConnection function.

MyJob::dispatch(); // will dispatch to default connection

//you should:
MyJob::onConnection('redis_a')->dispatch();
//or
MyJob::onConnection('redis_b')->dispatch();
// to dispatch on specific connections

On your database.php https://github.com/laravel/laravel/blob/8.x/config/database.php#L129 you should have the following:

'redis' => [
		...
        'redis_a' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', '6379'),
            'database' => env('REDIS_DB', '0'),
        ],
        'redis_b' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', '6379'),
            'database' => env('REDIS_DB', '0'),
        ],
		...
]

Of course, change it to your true credentials but you get the point.

And to have your redis_a becomes the default driver, change default to redis_a: https://github.com/laravel/laravel/blob/8.x/config/queue.php#L67

Then make sure you clear your configs :)

I noticed that I can still see jobs that I submitted to Redis B in Pending jobs of horizon of Application X Since you didn't define where you should dispatch your jobs, you dispatched all your jobs to the default connection.

tr8norman's avatar

hi @neilstee

Thank you very much for the very detailed reply!

I have actually done all the steps that you mentioned. But still I observed that I can still see jobs submitted to redis_b on my horizon which is supposed to be monitoring redis_a only.

Actually I realize that my question can be summarised by:

How do we set which queue connection, Horizon should monitor? Is it always the default queue connection only?

tr8norman's avatar

@neilstee ah ok I found

'use' => 'default',

in horizon.php. I think this is how Horizon know which Redis connection to use

neilstee's avatar

@tr8norman yes in horizon.php just change the use to point in your redis_a only connection.

Also, make sure it's really a different Redis instance, or else it will still read your jobs.

neilstee's avatar

What I mean is, maybe you have set up 2 connections but they really point to only one connection.

If you want to have 2 connections but live on the same Redis connection, you can modify the prefix. So redis_a connection only reads data with a specific prefix and the same with redis_b.

tr8norman's avatar

Hi @neilstee

I have another question related to this and I hope that you can help.

If I understand correctly for use, the explanation given in the config file is

This is the name of the Redis connection where Horizon will store the
    | meta information required for it to function. It includes the list
    | of supervisors, failed jobs, job metrics, and other information.

So this is actually not related to which Redis queue connection Horizon is listening to.

My hunch is that Horizon monitors job submitted to any Redis connections since Horizon basically overrides RedisQueue implementation which will be used regardless which Redis connection is used.

So in the case that an Application has 2 different Redis Connection (each have different Queue, Queue A and Queue B respectively), Horizon actually monitors jobs submitted to both Redis Connection.

Can you clarify if this is the case?

neilstee's avatar

@tr8norman

So this is actually not related to which Redis queue connection Horizon is listening to.

No, it will only get the jobs to that particular connection.

My hunch is that Horizon monitors job submitted to any Redis connections since Horizon basically overrides RedisQueue implementation which will be used regardless which Redis connection is used.

No, UNLESS you didn't configure connections to have different prefixes. https://github.com/laravel/laravel/blob/8.x/config/database.php#L126

Basically, you need to move that prefix key inside the default key or your new connection.

So in the case that an Application has 2 different Redis Connection (each have different Queue, Queue A and Queue B respectively), Horizon actually monitors jobs submitted to both Redis Connection.

No, as well.

You can understand the Redis prefix here more: https://laracasts.com/series/build-and-configure-a-staging-server/episodes/5

Please or to participate in this conversation.