Running Multiple Queue Workers

You could run queue as async with beanstalkd and use some sockets to callback a message to user after job is done.

Take a look here https://github.com/oriceon/laravel-5-sockets-and-queue-async

I've used the numprocs option in supervisor to run multiple queue workers. I use both multiple queue tubes as well as multiple workers on the queue tubes that benefit most from concurrency (file uploads, in my case).

If I am not mistaking, queue:listen will listen for any job on the selected queue tube and will keep your command line occupied whilst indefinitely doing so, queue:work will process only one job and give back command and queue:daemon will spawn a different process and will keep running in the background. The daemon option is the most CPU efficient, but you should keep an eye out for memory leakage. And don't forget to call php artisan queue:restart in order to load you latest code (it doesn't reinterpret PHP for every job like listen and work will).

@oriceon I'm currently using Redis and would prefer to not change it for now. But I have bookmarked the link, it will be useful on another project I'll start from scratch (this one was already on production when I started working on it, that's why I want to change as few things as possible).

@JeroenVanOort the guy that worked on the project before told me that running queue:listen was duplicating the process, making some of the jobs run twice (if running with two process). He said that running queue:work --daemon didn't make it run more than once, but he tested on another queue with different jobs. From my search and from what you said, this shouldn't happen even with queue:listen, do you have any clue why it ran twice?

Thank you for your answers :)

What queue driver was being used? I know jobs shouldn't run more than once using any driver but I only know for sure with beanstalkd, because that's the one I use.

@JeroenVanOort it uses Redis from AWS' ElastiCache

Ok, I figured out why the jobs were running more than once when running more than one queue:listen. I had to dig a lot on the source code since I couldn't find much about it anywhere, so here are some explanations of my finds.

First, let me explain how the 3 types of Queue commands work:

queue:listen -> this one will keep spawning a new Process (using Synfony's Process class) running queue:work, that's why it always get fresh source code and it will hardly crash, it runs purely on framework code, which is more tested and change less, if anything is wrong with your code, only the spawned process will crash and the control will come back to the queue:listen process. But it will always wait for the Process to complete before spawning a new one. If there is no job to run, queue:work will return, then it will be respawned again and again and again, until there is work to do. That's why this one uses more resources.

queue:work -> simple get the next job from your queue, runs it and quit

queue:work --daemon -> check if there is any job, if there isn't, it sleeps for some time and check again. When there is job, it just fire it, on the same process. That's why this is more bound to crash and you need to run queue:refresh (so when it finishes the current job, it stops running, then you can start "queue:work --daemon" again) to get the fresh source code.

Now to the reason of my jobs running more than once: With the Redis Driver, there are Three Queues: The main one, with the name you gave it or "default" A delayed queue, named "queue_name:delayed" And a reserved, called "queue_name:reserved"

The thing is, when pop is called by the queue:work (remember that queue:listen uses queue:work and the daemon option runs with some of the queue:work code), the Redis driver calls lpop on Redis, to get and remove the next job. If there is any job, it pushes it to the reserved queue, with the current time + the expire time, by default it's 60 seconds. And then it returns the job so queue:work can fire it, when you call $job->delete(), it's removed from :reserved. Before it executes the lpop, it checks for expired jobs on both :delayed and :reserved, and move them back to the main queue. This means that if my job takes more than 60 seconds to run, and I have more than one process, the second process will move it from :reserved to the main queue and get fire it. (This will keep happen, 3 process and a job that takes 120 seconds = 3 jobs running, 4 process and 180 seconds = 4 jobs, etc.).

I couldn't find anything in the docs about :reserved and :delayed queues, nor about the expire variable from the Redis Queue Driver. From what I looked, the other drivers don't work like this. From my understanding, this is a bug... either with the Docs for not making it clear that I can have overlapping jobs with the Redis Driver or with the code.

@henrique Having exact the same issue. Thanks for sharing this!

How did you solve it in your project? Have you switched from redis? Or have you just put in some hack to override the expiration timeout?

Agree about seeing it as a bug.. Have you submitted it anywhere yet?

@dogik I just added "'expire' => null," to the Redis queue config:

'redis' => array(
            'driver' => 'redis',
            'queue'  => 'default',
            'expire'  => null,
        ),

Keep in mind that using this method, if the job fails it will get stuck at the :delayed queue and you will need to manually move it to the normal queue for it to reprocess (or go to failed_jobs if it exceeds the attempts). You could also set the expire to a value higher than your longest running job.

I'm on a new project that will also use Queues and I'll probably use Redis because that's what I know how to use, I will probably make my own command to run the queue (instead of queue:work or queue:listen), this way I will have more control on how it handles failures and everything.

I think I was in a hurry and ended up not reporting it. If you report it, feel free to use any part of my explanation on the report :)

@henrique Hello henrique, did you end up writing your own queue:listen/work implementation ? I use phpredis but if fails when setting queue redis driver : https://laracasts.com/discuss/channels/laravel/php-artisan-queuelisten-throws-call-to-undefined-method-redistransaction-with-phpredis

So I was looking for a way to keep using the Queue::push which works totally fine and handle the jobs processing with a custom script but I don't really know where to start..

Looks like the current Redis queue laravel implementation does not provides job locking : https://github.com/laravel/framework/issues/8577

Let me know if you have something :)

@trompx Hello, the problem described at that github issue looks like the same as I described on this thread (not because of lock since Redis::pop is an atomic transaction, but because Laravel handles it this way), I have answered that Issue so more people looks into it and we might find better solutions than just changing the expire value :)

As for running the Queues, I only had to add this to my composer.json:

"predis/predis": "~1.0",

Inside "require" and run a composer update.

Hope this helps.

@henrique Thank you very much for trying to help. I tried your solution but when installing predis, I still get the error Call to undefined method Redis::transaction() unless in app.php I keep the default 'Redis' => 'Illuminate\Support\Facades\Redis', but in that case I cannot use phpredis for the other functionalities in my app. I guess I will build a worker which just runs my queues with predis and keep phpredis in the app.

@JeroenVanOort I know this thread is a bit old. But was just wondering how the numprocs option works.

e.g. if i have this:

[program:laravel_queue]
command=php artisan queue:listen --timeout=1800 --tries=5
directory=/var/app/current
stdout_logfile=/var/app/support/logs/laravel-queue.log
logfile_maxbytes=0
logfile_backups=0
redirect_stderr=true
autostart=true
autorestart=true
startretries=86400
EOB

Can I just put numprocs=3 in there? Or do I need a process_name as well?

I'm not sure, but trying probably won't hurt.

Hi @Ryk,

have you tried it?

I had issues just using numprocs. Adding this seems to be working:

process_name=%(program_name)s_%(process_num)02d

taken from: https://github.com/Supervisor/supervisor/issues/121#issuecomment-207699120

Hello, I have a system with a "import from XLS" option, xls contains lakhs or n number of emails, the user uploads the file, which is added to a Queue (Queue::push). My problem is that i have set a cron in the windows scheduler which verify the emails in the list, Queue takes a while to process the emails to verify using dns records and mxhosts and so people will have to wait a lot. Let's say it takes hours to verify each file and there are 10 files in the Queue, the last one to upload will need to wait of hours. Is it safe to run multiple commands? The amount of process will vary as people start to use it more. Is there any other approach to have it run concurrently?

Hello guys any body have any idea about how many numprocs=? we can have in our supervisor laravel queue ?

@Bibhudatta_sahoo, you can add as many as you think you'll need. Mind your server resources though

@timborder stumbled upon your post from google. Added that line and my supervisor started working again. Thanks!