In Horizon, Redis Job are idles : get paused longer than normal without starting processing

What is your current setup? It might be a misconfiguration. It also might be a code thing? You can actually delay jobs, so are you sure you're not doing that?

@bobbybouwmann To answer you :

• I am not using delay jobs. Also all jobs class have internal parameter $tries = 0 (to avoid any retry).

• On my local server on Windows (only one worker, in debug mode), i dont have such issue.

• QueueA is a queue with around 100 jobs too and sometimes 1 or 2 jobs disappear without calling JobProcessed or JobFailed.Logging showed me these ghost jobs can reappear in my QueueA (loaded with other jobs) a couple of hours later.

• 100 jobs done on QueueB. When all completed, another 100 job are injected in QueueB and then stuck on pause in Horizon dashboard for half an hour.

• I notice the same thing for a single long job (10min) running on queueV (below). When completed (JobProcessed listener called) , it calls another long job which get stuck in the queueV on pause for approx half an hour.

Is there any issue in configuration below :

My queue has 2 supervisors :

    'redis' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => '{default}',
        'retry_after' => 900,// 15 min, job will be released back onto the queue
        'block_for' => null,
    ],

    'redis-medium-running' => [
        'driver' => 'redis',
        'connection' => 'default',
        'queue' => 'default_medium',
        'retry_after' => 2700, // 45min 
        'block_for' => null,
    ],

And in my Horizon :

      'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default',
                'queue1',
                'queue2',
                'queue3',
                ],
            'balance' => 'false',// false,processes queues in the order they are listed in configuration.
            'processes' => 3,
            'tries' => 3,
        ],

        'supervisor-medium-running' => [
            'connection' => 'redis-medium-running',
            'queue' => ['default_medium', 
                'queueA',
                'queueB',
                'queueC',
                ],
            'balance' => 'false',
            'processes' => 25,
            'tries' => 1,
        ],

Thought in my jobs parameter $tries =1, i noticed that:

• 1st batch of Job in Queue B or C will complete in around 20min
• 2nd batch inserted will get paused around 25 min
• It happens Queue retry_after = 45min

It seems correlated...

If you put in more than 50 jobs that each take around 20 minutes you will have to wait 40 minutes to start the 51th job right? So in that case it might take really long before a job is finished!

I don't see anything strange in your setup though!

@bobbybouwmann : In fact by this following discussion : https://github.com/laravel/framework/issues/15370 ,

i understood tries=0 mean unlimited retry and tries=1 mean one try (and not one retry) Putting in Job Class tries=1 as in Horizon config partially solved the issue, even if it is not clear why. (Still I am wondering which parameter superseed the other one : Job 'tries' or Horizon 'tries' parameter)?

For ghost jobs: every time i restart Horizon by deploying a new version of app, it seems long job still in process are displayed in pause in Horizon and make orphaned (so i cannot make them disappear by cleaning the queue) and then come back to pollute my new job results.

It is frustrating : is there a way to tag Jobs so as to filter them out if result outdated?

Ok, 2 points on which Laravel documentation is not clear and helped to solved my issues :

• tries=0 mean unlimited retry and tries=1 mean one try (and not one retry)

• Job will trigger JobFailed event only after checking retry_after which means if :

  • timeout=60s, tries=1, retry_after=10min even if Job will timeout at 60s, the failed event will happen after 10min, when retry will check no more tries to do.

Last thing not answered : which parameter superseed the other one : Job class 'tries' or Horizon 'tries' parameter?It seems it is Job class 'tries' even if checking with htop in cli, this is Horizon 'tries' parameter which is displayed in processes