Redis latency with Redis cluster on Kubernetes

We recently moved all of our application infrastructure from ECS on Amazon Web Services to Managed Kubernetes on Digital Ocean. The biggest change with this migration was the move to a managed Redis cluster to be shared across multiple applications; versus a local installation of Redis within the application's container. While the experience has been mostly positive, we have experienced issues with the Laravel application connecting to the cluster and causing HTTP requests to take minutes to resolve or simply return 504 errors. The main issue is the fact that it is random. We cannot trace the cause to a specific K8 pod or a specific request. I have verified that I have followed all of the documentation on the Laravel site with regards to the key and queue naming conventions cache()->get("{products}"), but we are still getting some pretty slow requests.

I also recently installed NewRelic in the application and this is a brief screenshot of the metrics it gathered. You can see the majority of the request time was from the Redis connector:

I know this is a bit of a niche question, but it is proven a pretty big problem for our production environments. Any help would be appreciated.

Here is what my environment looks like:

Redis Client: phpredis

database.php

return [
    'redis' => [

        'client' => env('REDIS_CLIENT', 'phpredis'),

        'options' => [
            'cluster' => env('REDIS_CLUSTER', 'redis'),
            'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),
        ],

        'cluster' => env('REDIS_CLUSTER_ENABLED', false),

        'clusters' => [
            'options' => [
                'cluster' => 'redis',
            ],
            'default' => [
                [
                    'scheme' => 'tls',
                    'url' => env('REDIS_URL'),
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD'),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => env('REDIS_DB', '0'),
                    'timeout' => 15,
                    'read_timeout' => 60,
                    'persistent' => 1,
                ],
            ],
            'cache' => [
                [
                    'scheme' => 'tls',
                    'url' => env('REDIS_URL'),
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD'),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => env('REDIS_CACHE_DB', '1'),
                    'timeout' => 15,
                    'read_timeout' => 60,
                    'persistent' => 1,
                ],
            ],
        ],
]

LaryAI

3 years ago

Best Answer

Level 58

It sounds like you have configured your Redis cluster correctly, so the issue may be related to the latency of the cluster itself. To reduce latency, you can try increasing the timeout and read_timeout values in your database.php configuration. This will give the cluster more time to respond to requests. Additionally, you can try increasing the number of Redis nodes in the cluster to spread the load across more servers.

If the issue persists, you may want to consider using a different Redis client. For example, you could try using the predis client instead of phpredis. This may help reduce latency and improve performance.

1 like

dniccum

3 years ago

Level 2

This was the ticket as indicated by @laryai. Just in case anyone needs this, my DigitalOcean cluster configuration setup is below. Please note that I set the REDIS_SCHEME env variable to tls as this is required by DigitalOcean to connect to their clusters.

<?php

use Illuminate\Support\Str;

return [

    'redis' => [

        'client' => env('REDIS_CLIENT', 'predis'),

        'options' => [
            'cluster' => env('REDIS_CLUSTER', 'redis'),
            'prefix' => env('REDIS_PREFIX', Str::slug(env('APP_NAME', 'laravel'), '_').'_database_'),

            'parameters' => [ // Parameters provide defaults for the Connection Factory
                'password' => env('REDIS_PASSWORD', null), // Redirects need PW for the other nodes
                'scheme'   => env('REDIS_SCHEME', 'tcp'),  // Redirects also must match scheme
            ],
        ],

        'default' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_DB', 0),
        ],

        'cache' => [
            'url' => env('REDIS_URL'),
            'host' => env('REDIS_HOST', '127.0.0.1'),
            'password' => env('REDIS_PASSWORD', null),
            'port' => env('REDIS_PORT', 6379),
            'database' => env('REDIS_CACHE_DB', 1),
        ],

        'cluster' => env('REDIS_CLUSTER_ENABLED', false),

        'clusters' => [
            'options' => [
                'cluster' => 'redis', // This tells Redis Client lib to follow redirects (from cluster)
            ],
            'default' => [
                [
                    'scheme' => env('REDIS_SCHEME', 'tcp'),
                    'url' => env('REDIS_URL'),
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD'),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => env('REDIS_DB', '0'),
                    'timeout' => 15,
                    'read_timeout' => 60,
                    'persistent' => 1,
                ],
            ],
            'cache' => [
                [
                    'scheme' => env('REDIS_SCHEME', 'tcp'),
                    'url' => env('REDIS_URL'),
                    'host' => env('REDIS_HOST', '127.0.0.1'),
                    'password' => env('REDIS_PASSWORD'),
                    'port' => env('REDIS_PORT', 6379),
                    'database' => env('REDIS_CACHE_DB', '1'),
                    'timeout' => 15,
                    'read_timeout' => 60,
                    'persistent' => 1,
                ],
            ],
        ],

    ],

];

Please or to participate in this conversation.