Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

andyh81's avatar

Batches stopping when using redis

I created a csv uploader that chunks up the file by rows of 1000, then each new chunk is added to a batch for the import process. Everything works great and can easily upload files with 500k+ rows when using a database connection. When I tested out using redis, it will never finish the batch and usually just stop randomly between 5% to 80% jobs. I'm hosting on Cloudways and contacted support who were increasing the redis size to around 2GB. It didn't matter what size of csv I tried, it wouldn't even finish a file that had 10k rows. A reason I want to use redis is I can easily use Supervisord to monitor the queue and have to contact support to have them set it up to monitor a DB connection.

My question is should this be a process that only uses a database connection and not something redis would be good for or any thoughts on why it would be stopping when using redis?

0 likes
1 reply
LaryAI's avatar
LaryAI
AI
1 year ago
Level 58

When dealing with batch processing and Redis, there are a few considerations and potential issues that could cause the behavior you're experiencing. Here are some steps and suggestions to help troubleshoot and potentially resolve the issue:

  1. Check Redis Configuration:

    • Ensure that your Redis configuration is optimized for the workload. Redis is an in-memory data store, and if you're running out of memory, it could lead to jobs being dropped or not processed.
    • Check the maxmemory setting in your Redis configuration. If Redis runs out of memory, it will start evicting keys based on the configured eviction policy.

  2. Monitor Redis Usage:

    • Use Redis monitoring tools to check memory usage and keyspace information. This can help you identify if memory limits are being hit or if there are other issues with the Redis server.

  3. Increase Redis Memory:

    • If you have the ability to increase the memory allocated to Redis, try doing so. However, keep in mind that Redis is not designed to handle very large datasets that exceed available memory.

  4. Batch Size:

    • Consider reducing the batch size. While 1000 rows might work well with a database connection, it might be too large for Redis, especially if each job is memory-intensive.

  5. Job Timeout:

    • Check if there are any timeout settings for your jobs. If a job takes too long to process, it might be getting killed before completion.

  6. Queue Configuration:

    • Ensure that your queue worker configuration is set up correctly. This includes the number of workers, retry attempts, and any other relevant settings.

  7. Error Handling:

    • Implement robust error handling and logging in your job processing logic. This can help you identify if there are specific errors causing the jobs to fail.

  8. Redis Persistence:

    • If you're using Redis persistence (RDB or AOF), ensure that it's configured correctly. However, keep in mind that Redis persistence is not designed for durability like a traditional database.

  9. Consider Alternative Solutions:

    • If Redis continues to be problematic, consider using a different queue driver that might be more suited to your workload, such as a database or a dedicated queue service like Amazon SQS or RabbitMQ.

Here's a basic example of how you might configure a queue worker in Laravel to handle Redis:

// In your .env file
QUEUE_CONNECTION=redis

// In your queue worker configuration
php artisan queue:work redis --queue=default --sleep=3 --tries=3 --timeout=90

By following these steps, you should be able to identify the root cause of the issue and determine whether Redis is suitable for your use case or if another solution might be more appropriate.

Please or to participate in this conversation.