Multiple queues for Redis connection

Hello there.

I have this config file for queues

<?php

return [

    'default' => env('QUEUE_DRIVER', 'redis'),


    'connections' => [

        'sync' => [
            'driver' => 'sync',
        ],

        'database' => [
            'driver' => 'database',
            'table' => 'jobs',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'beanstalkd' => [
            'driver' => 'beanstalkd',
            'host' => 'localhost',
            'queue' => 'default',
            'retry_after' => 90,
        ],

        'sqs' => [
            'driver' => 'sqs',
            'key' => 'your-public-key',
            'secret' => 'your-secret-key',
            'prefix' => 'https://sqs.us-east-1.amazonaws.com/your-account-id',
            'queue' => 'your-queue-name',
            'region' => 'us-east-1',
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
            'queue' => 'default',
            'retry_after' => 90,
        ],

    ],

    'failed' => [
        'database' => env('DB_CONNECTION', 'mysql'),
        'table' => 'failed_jobs',
    ],

];

It is almost the default config/queue.php file generated by Laravel. To run my redis queue, I've made a Linux service script that runs this command:

php artisan queue:work redis --sleep=3 --tries=3 --daemon

Usually I use it to process a email notification system, but now I need to use the queue system to process some command line functions and I want to use a separate queue. I'm kinda new in queues and I couldn't find any information about how can I create multiple queues for the same connection. Instead, I've found people using different connections for each different queue. Do I really need to configure it that way in my config/queue.php?

This is something that is missing in laravel documentation and I cannot find out another solution. It seems weird to me if I have to configure the same connection multiple times with different names in the connection name and queue. Cannot I use an array in the queue parameter for a connection?

osteel

7 years ago

Level 1

Hey,

That's an old question but I'm gonna answer anyway as I was wondering the same thing and couldn't find the answer anywhere.

I've seen everywhere people declaring multiple otherwise identical connections to achieve running multiple queues, like so:

'connections' => [

    'redis_high' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'high',
        'retry_after' => 90,
        'block_for' => null,
    ],

    'redis_default' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => 90,
        'block_for' => null,
    ],

]

This feels odd as Laravel's documentation insists itself on the fact that a same connection can run multiple queues.

So I went on testing for myself and I found out that you actually don't need to specify each queue if you want to use the same connection. Say you declared this connection:

'connections' => [

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default',
        'retry_after' => 90,
        'block_for' => null,
    ],

]

And that Redis is your default driver. If you push a job without specifying a queue, it will be pushed on the default one:

Queue::push($job);

If you specify a queue however, it will be pushed onto it even if it does not appear in the queue.php config file:

Queue::pushOn('high', $job);

As long as you specify that queue in the Supervisor config, its jobs will be processed:

command=php /var/www/site/artisan queue:work redis --queue=high,default

So it looks like it's optional to have them in the config file as long as a default connection is set up. I guess the only interest to list them all in queue.php is to keep track of them, or if indeed you need specific settings for a particular queue (i.e. you need to declare a separate connection).

14 likes

qwertynik

5 years ago

Level 5

Thanks for the detailed answer :) @osteel

johnvoncolln

7 years ago

Level 8

Thanks - been searching around and couldn't find anything definitive until I saw your post!

oscar_g__

6 years ago

Level 2

How does Laravel know the queue "high" is actually a high priority? How would you assign a priority to a queue if you gave it a name other than "high" or "default" ?

seanmtaylor

6 years ago

Level 1

The priority is determined by the order they are passed into the ---queue option.

So if your highest priority queue was called foo, you would do:

php artisan queue:work --queue=foo,default

That would give the foo queue higher priority than your default one.

But if you wanted the default queue to be higher priority then you would do:

php artisan queue:work --queue=default,foo

https://laravel.com/docs/7.x/queues#connections-vs-queues

Some applications may not need to ever push jobs onto multiple queues, instead preferring to have one simple queue. However, pushing jobs to multiple queues can be especially useful for applications that wish to prioritize or segment how jobs are processed, since the Laravel queue worker allows you to specify which queues it should process by priority. For example, if you push jobs to a high queue, you may run a worker that gives them higher processing priority:

php artisan queue:work --queue=high,default

Please or to participate in this conversation.