1 central Redis host for multiple client servers with Laravel application

I have a multi server environment on which a Laravel application runs. There are 5 application servers and 1 database server that also has Redis installed and acts as central Redis host for the other application servers. All servers are Ubuntu 22 OS servers.

I configured the 5 application servers all the same to use Redis driver for cache, sessions and queues. Redis configuration all point to the same host server and Horizon is installed with Supervisor to keep everything running.

Now I have this weird problem. When I check Horizon dashboard on each of the application servers then not all supervisor processes of each server are visible. I have had different combinations like

servers 1, 3 and 4 are visible together and 2 and 5. This means in Horizon dashboard of server 1, 3 and 4 you can only see supervisor processes 1, 3 and 4. And 2 and 5 on servers 2 and 5.
servers 1, 3, 4 and 5 and 2 as single process.

I have noticed that in these situations the cached data is available in the same combinations. So for the situation where the servers are grouped in 2 servers and 3 servers then the data that was first cached on one of the 2 servers is not available on one of the 3 servers. Only after the data is also cached separately for the other group as well.

I have checked a lot of the configuration in the Laravel application and the server as well but it kind of feels random why there is seperation and not all servers are handled together. All servers can connect and they all can store and read data and process queued jobs, only thing is that not everything is visible/available on each application server.

So to be clear. My goal is that ALL data for cache, sessions and queues that is processed through Redis in the Laravel application on ALL 5 servers is availabe at the same time for ALL application servers. And I want to use this central Redis host server for this.

Is there someone who can help?

Thanks!

LaryAI

1 year ago

Level 58

To address the issue of inconsistent visibility of supervisor processes and cached data across your application servers, you need to ensure that all servers are correctly configured to use the same Redis instance and that there are no network or configuration issues causing the separation. Here are some steps to troubleshoot and resolve the problem:

Check Redis Configuration: Ensure that all your application servers are pointing to the same Redis host and port. Verify the configuration in your config/database.php and config/cache.php files.

// config/database.php
'redis' => [
    'client' => env('REDIS_CLIENT', 'phpredis'),
    'default' => [
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD', null),
        'port' => env('REDIS_PORT', 6379),
        'database' => env('REDIS_DB', 0),
    ],
],

// config/cache.php
'stores' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
    ],
],

Ensure that the .env file on each server has the correct Redis host and port:

REDIS_HOST=your_redis_host
REDIS_PORT=6379

Network Connectivity: Verify that all application servers can connect to the Redis server. You can use tools like telnet or redis-cli to test connectivity.
```
telnet your_redis_host 6379
```
Or using redis-cli:
```
redis-cli -h your_redis_host -p 6379 ping
```
You should receive a PONG response if the connection is successful.
Horizon Configuration: Ensure that Horizon is correctly configured to use the same Redis connection on all servers. Check your config/horizon.php file.
```
'use' => 'default',
```
Also, ensure that the Horizon configuration in the .env file is consistent across all servers.
Supervisor Configuration: Ensure that the Supervisor configuration files are correctly set up on each server. Each server should have a unique identifier for its Horizon processes. You can set this in the horizon.php configuration file:
```
'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default'],
            'balance' => 'auto',
            'processes' => 10,
            'tries' => 3,
        ],
    ],
],
```
Ensure that each server has a unique supervisor name (e.g., supervisor-1, supervisor-2, etc.).
Cache and Session Configuration: Ensure that the cache and session configurations are consistent across all servers. Check config/session.php and config/cache.php.
```
// config/session.php
'driver' => env('SESSION_DRIVER', 'redis'),

// config/cache.php
'default' => env('CACHE_DRIVER', 'redis'),
```
Clear Cache: Clear the application cache and config cache on all servers to ensure that there are no stale configurations.
```
php artisan config:cache
php artisan cache:clear
php artisan horizon:terminate
```
Check Redis Persistence: Ensure that Redis persistence is correctly configured to avoid data loss. Check your Redis configuration file (redis.conf) for appendonly and save directives.
```
appendonly yes
save 900 1
save 300 10
save 60 10000
```

By following these steps, you should be able to ensure that all your application servers are correctly configured to use the central Redis host and that all data for cache, sessions, and queues is available across all servers. If the problem persists, consider checking for any network issues or Redis server performance issues that might be causing the separation.