Oct 22, 2023
0
Level 1
Redis repeatedly saves the queue database (horizon) resulting in loss of pending jobs
When we used Redies for queued jobs we faced the following problem: Redis constantly overwrites the database almost every second. Here's the log:
30:C 21 Oct 2023 09:24:21.091 * DB saved on disk
30:C 21 Oct 2023 09:24:21.091 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 21 Oct 2023 09:24:21.189 * Background saving terminated with success
1:M 21 Oct 2023 09:25:22.045 * 1 changes in 60 seconds. Saving...
1:M 21 Oct 2023 09:25:22.045 * Background saving started by pid 31
31:C 21 Oct 2023 09:25:22.048 * DB saved on disk
31:C 21 Oct 2023 09:25:22.049 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 21 Oct 2023 09:25:22.146 * Background saving terminated with success
1:M 21 Oct 2023 09:26:23.018 * 1 changes in 60 seconds. Saving...
1:M 21 Oct 2023 09:26:23.018 * Background saving started by pid 32
If a job is delayed for long time (like 30 minutes) it is just erased from the queue. We ran the app in Docker with following container settings:
redis:
image: redis:alpine
ports:
- "6379:6379"
command: redis-server --appendonly yes --appendfsync everysec
restart: unless-stopped
volumes:
- selenium_redis:/data/
We set --maxmemory to 1024mb and more but it helps only for small amount of time and then the looped overwriting starts again.
We have changed the queue driver which solved the problem, but now we are curious: what is the issue with Redis?
Please or to participate in this conversation.