Redis as cache & queue driver with compression

Hello!

I just killed production app tonight, by activating compression on my redis cluster. I'm using redis for 3 things : cache, session, queue (horizon).

I've just added

'compression' => Redis::COMPRESSION_LZ4,

to my config. It worked well for my cache and session, but after a few hours, I realized queued emails were not sent anymore. After hours of debugging I realized that the doc mentionned on the queue page

The serializer and compression Redis options are not supported by the redis queue driver. https://laravel.com/docs/12.x/queues#connections-vs-queues

Okay, so my question is : is there a way to say "use compression for cache and not for queue" ? Having to cache my entire site uncompressed is because of this is unreasonable (i currently have 15gb of cached data for my site). From what i've tested, I could go down 3-4gb using the compression.

Not using Horizon is not an option, so if this is not possible I may just have to use something else for caching.

I've surprisingly found nothing to this point to help, AI is mainly suggesting broken things, so maybe a human will help!

Thanks

LaryAI

6 months ago

Best Answer

Level 58

You're right in noticing that Laravel does not support using Redis compression (or custom serializer) options with the redis queue driver (see the official docs). However, you can work around this limitation by using separate Redis connections: one with compression for your cache/session, and one without compression for your queue.

Here's how you can set this up:

1. Define Two Redis Connections

In your config/database.php, under the 'redis' => [ ... ] array, define two connections:

'redis' => [

    'default' => [
        'url' => env('REDIS_URL'),
        'host' => env('REDIS_HOST', '127.0.0.1'),
        'password' => env('REDIS_PASSWORD'),
        'port' => env('REDIS_PORT', '6379'),
        'database' => 0,
        'compression' => Redis::COMPRESSION_LZ4, // Enable compression here
    ],

    'queues' => [
        'url' => env('REDIS_QUEUE_URL', env('REDIS_URL')),
        'host' => env('REDIS_QUEUE_HOST', env('REDIS_HOST', '127.0.0.1')),
        'password' => env('REDIS_QUEUE_PASSWORD', env('REDIS_PASSWORD')),
        'port' => env('REDIS_QUEUE_PORT', env('REDIS_PORT', '6379')),
        'database' => env('REDIS_QUEUE_DB', 1),
        // **Do NOT specify 'compression' here!**
    ],
],

2. Configure Each Service to Use the Proper Connection

Cache (and session)

In your config/cache.php (and config/session.php), set the Redis connection to default (the compressed one):

'stores' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
    ],
    // ...
],

Queue

In your config/queue.php set the Redis queue connection to queues (the uncompressed one):

'connections' => [
    'redis' => [
        'driver' => 'redis',
        'connection' => 'queues',
        // ...
    ],
    // ...
],

3. (Optional) Use Separate Databases

If you want a full separation, use different databases for default and queues connections, as shown above (database => 0 and 1).

4. Restart Your Services

After making these config changes, clear your config cache and restart relevant services:

php artisan config:cache
php artisan horizon:terminate

(or restart your queue worker process)

Summary

Use a compressed Redis connection for cache/session.
Use an uncompressed Redis connection for queues.
Point your Laravel configs appropriately.

This way, you get the best of both worlds: compressed caching and working queues!

Let me know if you need example .env config or deeper setup!