Job vs. Horizon Timeouts

Hello! I'm trying to understand the difference between the timeout in a Job Class and the timeout in the horizon config. Could someone either explain the difference or point me to a something that explains the difference?

I've pasted a simplified version of my setup below. I previously had a long-running job fail silently and realized that it started finishing when I added the horizon timeout key/value. However, I don't understand why the job timeout alone is insufficient.

(I've confirmed I'm dispatching that job on the long-running queue)

Job Class:

public $timeout = 3600;

protected $report;

public function __construct(Report $report)
{
    $this->job = $report;
}

public function handle()
{
    $this->report->process();
}

Horizon Config:

        'production' => [
            'jobs' => [
                'connection' => 'redis',
                'queue' => env('REDIS_QUEUE', 'default'),
                'balance' => 'auto',
                'processes' => 8,
                'tries' => 1,
                'memory' => 128,
                'timeout' => 90,
            ],
            'long-running-jobs' => [
                'connection' => 'redis-long-running',
                'queue' => env('REDIS_QUEUE', 'default') . '-long-jobs',
                'balance' => 'auto',
                'processes' => 2,
                'tries' => 1,
                'memory' => 128,
                'timeout' => 3615,
            ],
        ],
    ],

klopma

4 years ago

Level 3

In case this helps anyone else (looking at you, future self) or, better yet, if anyone else would like to clarify / correct me, my conclusion was essentially that:

The job itself has a timeout trigger so that it'll end itself if it's running too long. You can change that by setting $timeout on the job class itself.
The queue workers are also supposed to look at a job and say "this has been running too long, end it." That value is also defaulted to 60 seconds and has to be overwritten. While using Horizon, you set that value for queue workers in the horizon config.

In case this is also helpful for clarity:

There's also some confusion with retryAfter on the job class and retry_after in the queue config. My understanding is:

retryAfter in the job class indicates how long a (known) failed job should wait until it tries again. So if a job decides that it has failed, it'll wait [$retryAfter] seconds before firing off again (assuming the number of tries is > 1).
retry_after in the queue config is a supervisor-specific setting that says "Okay, I sent that job off to be worked on, [retry_after] time has passed and I still haven't heard anything back. Let's just assume it died and fire off a new one.

Assuming that's true, I would assume your setup should have:

$timeout (on the job class) < timeout (in the horizon config) < retry_after

Otherwise the job will either be killed prematurely or a second instance will be fired off when we haven't yet given the first enough time to finish.

Hope that helps! And/or if it doesn't (or I'm just wrong) please reply below so I'm not spreading misinfo!

3 likes

NickVahalik

1 year ago

Level 8

This helped me.

Please or to participate in this conversation.