How to dynamically generate queues in Laravel Horizon

Well you can dynamically set the config on a request right? You can probably do that in a middleware or a service provider.

Note: I know that the middleware way works, but I'm not sure if the service provider way works

Here is an example for setting all the configs per user.

public function handle()
{
    $users = User::all();

    foreach ($users as $user) {
        Config::set('horizon.environments.production.user-'. $user->id, [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'simple',
            'processes' => 10,
            'tries' => 3,
        ]);
    }
}

Note that I'm not a big fan of this. Handling a queue per user sounds like a bad idea and a waste of resources. My advice would be to have a queue per domain of your site. So for example a queue for emails, a queue for processing images and a queue for the other stuff.

Thanks for your reply :) You said that "I'm not a big fan of this" what could be other solutions ? In my case I have thousands of jobs for every user and these jobs should be processed one by one for every user

The problem with the current configuration is that you probably end up with hundreds of workers that do nothing, if all tasks for a specific user are done. they eat memory and cpu and can not do something else on other busy queues, so you loose the power of the balance strategies.

I think it would be better you have X processes working on Y queues, so just append to the queues array instead of defining 1000 separate worker configurations.

'supervisor-1' => [
    'connection' => 'redis',
    'queue' => ['default', 'emails', 'user-1', 'user-2', 'user-3', 'user-4'],
    'balance' => 'auto',
    'processes' => 100,
    'tries' => 3,
],          

Now you should have 100 processes that should jump back and forth between the queues, wherever there is much to do.

Well like I said. I would probably use a handful of queues that would handle the related tasks for a user. It can be any user at that point. A queue is only there to postpone tasks that are not necessary for the current request. If you have a lot of jobs for later usages you should have enough resources for those queues and let them handle 100 processes at once. You will need a big server for that, but it's possible ;)

Well the reply from @Helmchen is exactly what I mean! Run multiple processes on x number of queues. You can of course add an extra server (supervisor-2), but first start with multiple processes and queues.

Thanks you all for your kind responses but if I set balance to auto user-2 can have 30 processes but it shouldnt be more than 1.

Do you have a solution for this?

And can I add dynamically queues array like the solution in middleware method?

Thanks a lot

I would probably leave it up to the job to do it in parallel or not. With thousands of jobs per user you break your neck with the limitation to 1 worker per queue.

Redis has some nice concurrency features in Laravel, which I am also looking at myself.

https://laravel.com/docs/5.6/queues#rate-limiting

Although I'm not sure how the performance looks like if you really have thousands of jobs per user.

@snare did you found a solution?