How does one utilise multiple Queue/Worker Servers in a Load Balanced Environment?
I recently watched a create tutorial by @Fideloper on his serversforhackers.com site, which details how to Load Balance your Laravle application using Forge.
In it, he touched upon the possibility to set up as many Worker/Queue servers as you need. As my application is very intense, I wanted to add a second one.
Is it as simple as setting up the first? So, I install my application on another server, and set up the Queues?
How would the two servers behave? Can they interfere with each other?
Is it as simple as setting up the first? So, I install my application on another server, and set up the Queues?
Yes!
You have some options:
If your server resources are under-utilized, you can add more workers to your one server (multiple things running php artisan queue:work ..., or if you're using supervisor, increase the number of processes in the configuration).
If you need more workers but can't run them all on one server because of server resource usage, you can use multiple servers.
Caveats to multiple servers:
All servers need access to the same database instance (and likely same cache if using redis/memcached). This likely means making sure the 2nd server can reach the database on the first server. Forge can help you out there, it has settings for allowing connections to/from each server on certain ports.
On the 2nd server, assuming it's only used for workers and not for accepting web requests, you can disable nginx so it won't accept web requests (or even use the ufw command to disallow HTTP(S) connections on port 80/443).
Great @fideloper this is very helpful. How does the queue load then get shared though? Maybe I am being a bit stupid. Is it a manual thing, I would ensure that certain jobs get directed to one queue and other jobs to the second queue?
It's not like the system will handle this itself, by balancing load across the two queue servers?
So should I configure it so one server runs say the HIGH queue items, and the other the LOW/MEDIUM and less urgent ones? Would that even help?
Ideally, it would be good if the second queue server could pick up some of the HIGH/URGENT load to speed it up.
