How can we scale laravel queue using redis?

We are using laravel queue in Redis and I can see that the queue is saved into one Redis key. so how to scale this implementation?

I know that I can create multiple queues with multiple workers, but can we split the redis queue between multiple Redis keys for the same queue, or is that hard to do?

thanks in advance.

crnkovic

4 years ago

Level 43

I think you should examine would splitting this up actually help you scale?

Redis is blazing fast and from my past experience on scaling, I'd focus on more queues, more workers, more CPU power and more processes for each worker and of course optimizing jobs to run as fast as possible. I think this will help you much more than overriding how Laravel actually stores data in Redis.

engahmeds3ed

4 years ago

Level 1

Many thanks for your reply, So do you think there is any other cache driver other than redis can work faster and we can scale it horizontally ( maybe sqs ). We have millions of queue items daily.

bugsysha

4 years ago

Level 61

You should explain why your queue is not scalable.

Did you follow the best practices of scalable queues?

If a task is interrupted, it can be restarted and completed successfully
A task can be called multiple times, without changing the side effects
More than one task can be run at the same time
The order of the tasks does not matter

engahmeds3ed

4 years ago

Level 1

Thanks for your reply.

We think that the Redis queue isn't scalable horizontally as the queue itself is inside one Redis key so when adding a new Redis server and use Twitter Twemroxy as a proxy to handle data between both servers, the queue always live at only one Redis server and the other doesn't have any queue inside it.

Also now we have a problem that the queue is going larger and larger without removing the older ones and we don't know why?!

regarding those best practices, from where can I get more info about them?

Thanks again for your time.

bugsysha

4 years ago

Level 61

We think that the Redis queue isn't scalable horizontally

I would try to optimize the jobs that are running because I see a bunch of developers reach out too early to scale horizontally.

Also now we have a problem that the queue is going larger and larger without removing the older ones and we don't know why?!

Exactly my point. You are trying to fix the consequence, and not the problem itself.

regarding those best practices, from where can I get more info about them?

Those best practices are not something that I've found on the internet. It is my opinion on how jobs should be written to be scalable. I guess there are more, but those are the key ones that I follow always.

Thanks again for your time.

You are very welcome.

engahmeds3ed

4 years ago

Level 1

Many thanks again, your reply is very valuable to me.

May I ask you a question plz?

if I have queue items being handled properly but then added to "queues.QUEUE_NAME:reserved" key in redis and not going out of it, what may cause this issue?

Please or to participate in this conversation.