DockerSwarm/Kubernetes - How configure Horizon

When you scale your Laravel application that uses Horizon across multiple containers in Docker Swarm or Kubernetes, Horizon is designed to work with such distributed systems. However, there are a few considerations to ensure that jobs are processed correctly without duplication.

Horizon uses Redis to manage job processing, ensuring that a job is only handled by one worker at a time. When you scale your application, you are essentially adding more workers that can pick up jobs from the Redis queues. Here's how you can ensure that your jobs are handled properly:

1. Unique Jobs: Make sure that your jobs are idempotent and can be uniquely identified. This way, even if a job is picked up by multiple instances, it will only be processed once.

2. Job Locking: Horizon supports job locking to prevent the same job from being processed by multiple workers. You can use the withoutOverlapping method in your job chain to prevent job overlap.

3. Supervisors Configuration: Configure your Horizon supervisors to distribute the workload evenly. You can define multiple supervisors with different balancing strategies, such as simple, auto, or false.

4. Monitoring: Use Horizon's dashboard to monitor your queues and workers. This will help you identify if multiple instances are trying to handle the same job.

5. Scaling Strategy: When scaling, consider using a strategy that fits your workload. For CPU-intensive jobs, you might scale based on CPU usage. For I/O-bound jobs, you might scale based on the length of the job queues.

Here's an example of a Horizon configuration (config/horizon.php) with multiple supervisors:

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['high', 'default', 'low'],
            'balance' => 'auto',
            'processes' => 10,
            'tries' => 3,
        ],
        'supervisor-2' => [
            'connection' => 'redis',
            'queue' => ['high', 'default', 'low'],
            'balance' => 'auto',
            'processes' => 10,
            'tries' => 3,
        ],
        // ...
    ],
    // ...
],

In summary, when you scale your Laravel application with Horizon across multiple containers, Horizon and Redis work together to ensure that each job is processed only once. Make sure to configure your supervisors and job handling strategies appropriately to handle the increased load. Monitor your system to ensure that jobs are being processed as expected without duplication.