How to speed up queues?

I've made an app that does both; it splits up different kinds of jobs and can process some types in parallel.

But please elaborate on the current performance. What does it need to do and what time does it take?

We are using redis as queue for now. But the work is a variety of work, some is fast, some is slow - but the challenge is we dump 10,000 items in the queue and with one worker if the worker has to wait for outside connection then the whole queue is clogged up.

Which one do you think is giving you the better performance?

I guess the face that you have done it is good enough for me to run down that path. I am learning towards 9 queues and 9 listeners.

@JeroenVanOort, can i see your github for queues. Would love to read and understand source.

If you want to speed up your queue processing, the first step is to add more queue workers. You could do that whether you split things into multiple queues or not.

You might also consider trying beanstalkd, it's got very low overhead, and very low latency. It was specifically built to be a queue, and only a queue. Unlike redis which can be a queue. beanstalkd has one job, and it does that job very well.

@kodeine I'm sorry, it's not open source since it's the core business software for a customer of mine.

And as @willvincent said: use multiple workers. I don't know about beanstalkd vs redis queue performance, but I do know beanstalkd just works.

Anyone know specifically how to setup multiple workers?

Probably the best solution would be to install (supervisord)[http://supervisord.org/]. Not only will that let you launch multiple workers, but it will keep them running, can be set up to start them on boot, and defining the number of workers is a simple matter of setting a value in the config.

@JeffreyWay covers supervisor in this video: https://laracasts.com/lessons/supervise-this the config starts about 2min into the video.

He doesn't configure it for multiple workers though. To do that, your include file would look something like this:

[program:queue_runner]
command = php artisan queue:listen --tries=2
directory = /var/www
stdout_logfile = /var/www/app/storage/logs/supervisor.log
redirect_stderr = true
numprocs = 12
process_name = %(program_name)s%(process_num)s

That would launch 12 queue workers.

See full config docs here: http://supervisord.org/configuration.html

No meaning to hijack the topic, but what about system performance. i'm in the exact same position. my queue processes images into various thumbnails and such. so when you upload a huge gallery this can be time consuming but what i notice is that my system is getting clogged (cpu mainly). Would a queue system be smart enough to add processes if system load lets it?

@kayintveen you could set a nice level (process priorty) when launching your queue worker.

Process priority on *nix systems ranges from -20 to 20, with lower values being higher priority. by default most things launch with a priorty of 0.

So, if your queue is doing large amounts of work that could hog a lot of cpu time, setting it to a lower priority will allow it to get work done, but not prevent other tasks from running as expected. However, since you'd be bumping the priority way down, chances are good that queue items will take longer to process, so you should include a different timeout value than the default 60seconds.

Something like this should work well:

nice -n 19 php artisan queue:listen --tries=3 --timeout=300

Thats great, but thats not possible for iron.io or something

I am trying to match records in database and to match 200 records with another set of 200 is taking 5 minutes . Thats too slow . Although I want to do that with record set of 200000. I am trying to run 200 queues at the same time for that large dataset but it takes days to finish that. Any suggestions to speed that up

@suppal, it takes 5 minutes to match 200 RECORDS? That seems like something a single sql query should be able to do in milliseconds. If your queue is doing it one at a time I could see that being the case, but your queue could do batches at a time if that's necessary.