Multi-server Queues - Is Redis and Supervisor a good combination?

Sounds like you want to redistribute the queue workers across multiple servers instead of just one, correct?

I'm not sure what your current setup is - having a worker run as a supervisord jobs is pretty normal, and restarting the job might make sense if it's set to exit when it finishes running, but that depends on which variety of queue:work you may be using in Laravel 5.1

In any case, any queue worker system can do this well - I would suggest using something like beanstalkd over just the default "database" queue system (or no system at all), since queue systems help prevent jobs from getting pulled more than once (and other timing issues) which is important when using more than one worker.

In that setup, you have a single instance of beanstalkd running, and it's responsibility is to handle out jobs to as many workers as you decide to run.

One thing I often use instead of beanstalkd is SQS, since it's pretty easy to setup and usually only costs pennies. Having one less thing to manage is always nice!

One minor drawback to SQS is it's use of "visibility timeout", where once a job is taken, it has x seconds to complete (you can set that yourself, up to 12 hours I believe) - if your jobs take a longer than that time, then the job becomes visible again. It's important to set that value for each job, otherwise the job may become visible again and you may end up with more than one worker churning on it.

Thanks for responding @fideloper !

The plan is indeed to distribute my queue workers over several servers.

I am currently trying to do this by adding all of my jobs to a single Redis server and into a single queue. This single server sits behind two load balanced web servers, both of which use Redis to manage their sessions, cache and queues. Here is a snippet of my .env.example file:

CACHE_DRIVER=redis
SESSION_DRIVER=redis
QUEUE_DRIVER=redis

REDIS_HOST=**.**.**.**
REDIS_PASSWORD=******
REDIS_PORT=6379

My initial plan was to have workers on all three of the servers listening for jobs in this single queue as each server is capable of carrying out any of the jobs i am queuing (mainly writing to the database).

I will look into SQS a bit more as I haven't really had a play around with it yet.

Just for posterity, this style of redis/ supervisorsetup does indeed work!

Turns out that my issue actually stemmed from my /etc/supervisor/supervisor.conf and /etc/supervisor/conf.d/laravel-worker.conf files being slightly miss configured.

In laravel-worker.conf i had accidentally added the --daemon argument as part of the "command" line:

command=php /var/www/<project>/artisan queue:work redis --sleep=3 --tries=3 --daemon

This was preventing the worker from restarting

In supervisor.conf I had added passwords to the [unix_http_server] block on some servers but not others:

[unix_http_server]
file=/var/run/supervisor.sock    ; (the path to the socket file)
chmod=0700                              ; sockef file mode (default 0700)
username=user                           ; The username to access the unix interface
password=pass                           ; The password to access the unix interface

For the sake of testing, I removed this requirement and everything started working after i ran service supervisor restart