High Throughput Queues - Laravel Queue + Twemproxy?

Hey everyone!

We are currently dealing with an interesting scaling issue. We process a LOT of jobs through the queue every day (10s of millions, with upwards of 100s of thousands running at once). It seems like we are currently pushing the bounds of what a single, dedicated job Redis instance can process and are looking for solutions as our needs continue to grow.

Long story short - We are CPU bound on the job Redis server and are looking to get multiple Redis instances going to allow multiple CPU cores to be working on processing job data. Has anybody setup twemproxy and used it with multiple Redis instances for Laravel's Queue data? Is that even possible? I've spent a fair bit searching, but seem to be coming up empty handed.

If that isn't possible, is there a good way of splitting jobs, at the queue level, into different Redis connections to leverage additional CPU cores?

A few things we have already done:

split off each Redis instance into a dedicated server (cache, session, job, and misc data)
upgraded all dedicated Redis instances to Digital Ocean's high CPU servers
made sure each Redis instance is just running on db 0 in preparation for twemproxy

tmyers273

8 years ago

Level 9

Some updates for anyone interested. I found some time to play around with this some more today.

Setup a dedicated redis box, stood twemproxy up in front of it, and have been testing it with a local Laravel install. It appears that there won't be any issues running all of the queue jobs through that setup. The LUA scripts and associated calls all appear to be running correctly in this setup.

Getting twemproxy up in front of all of our production Redis instances should fix the intermittent (but regular) TOO MANY CONNECTIONS errors we have been getting.

This begs the question, is it preferable to have an instance of twemproxy on each of the worker servers? Or a single instance on the Redis server itself?

Next up will be getting multiple Redis instances going on a single machine to make better use of the available CPU cores. The jobs server will likely need some special care to ensure that the persistent hashing is setup correctly so that normal jobs, delayed jobs, etc are all on the same box.

Perhaps something can be done with the colons that Laravel uses to "namespace" jobs? If the underlying Queue doesn't have to be monkeyed with, that would be a preferred.

tmyers273

8 years ago

Level 9

In the process of doings this, I created a simple setup script for Redis 4.0. This installs 4.0, sets up basic settings, and prompts to be bound to the servers public and private ip. This is specifically setup for Digital Ocean due to how the public and private ips are gathered.

https://gist.github.com/tmyers273/95c47ec7887c7e791bdb3462f26ee1a6

Eventually hoping to turn this into something more full fledged to help with installs and setup of twemproxy as well as the setup of additional Redis instances on the same server. But for now, it's essentially just an install + sane set of defaults.

tmyers273

8 years ago

Best Answer

Level 9

Got it! Twemproxy with Laravel Queues IS possible. And it's not that hard!

Setup redis, twemproxy, and additional instances. I made a little helper script for that here https://github.com/tmyers273/friac
Test connectivity from a local machine
Setup a twemproxy queue driver. The main thing we are doing here is altering the naming of the queue to use hash tags for consistent hashing. So queues:my-queue:reserved turns into queues:{my-queue}:reserved
1. In config/queue.php change the driver to twemproxy
2. Config your job database connection to use the new host and port
3. Create the twemproxy driver (https://gist.github.com/tmyers273/f792ebae9772130fe8f16e4711e37245)
Deploy to production
Enjoy the new twemproxy setup!

Please or to participate in this conversation.