Keep jobs longer on Horizon before cleanup & better queueing

I have been having some failed jobs due to files being large mainly and things taking too long at Digital Ocean Spaces

Illuminate\Queue\MaxAttemptsExceededException · App\Jobs\ProjectBackupJob has been attempted too many times or run too long. The job may have previously timed out.
Raw
vendor/laravel/framework/src/Illuminate/Queue/Worker.php:755 Illuminate\Queue\Worker::maxAttemptsExceededException
     */
    protected function maxAttemptsExceededException($job)
    {
        return new MaxAttemptsExceededException(
            $job->resolveName().' has been attempted too many times or run too long. The job may have previously timed out.'
        );
    }

and failed job in table

{"uuid":"46ab338d-c36d-4255-b47f-889f04a5c995","timeout":null,"tags":["App\Models\Editor\Project:527"],"id":"46ab338d-c36d-4255-b47f-889f04a5c995","pushedAt":"1668301415.5834","backoff":null,"displayName":"App\Jobs\ProjectBackupJob","maxTries":null,"type":"job","failOnTimeout":false,"maxExceptions":null,"retryUntil":null,"data":{"command":"O:25:\"App\Jobs\ProjectBackupJob\":2:{s:34:\"\u0000App\Jobs\ProjectBackupJob\u0000project\";O:45:\"Illuminate\Contracts\Database\ModelIdentifier\":5:{s:5:\"class\";s:25:\"App\Models\Editor\Project\";s:2:\"id\";i:527;s:9:\"relations\";a:0:{}s:10:\"connection\";s:5:\"mysql\";s:15:\"collectionClass\";N;}s:32:\"\u0000App\Jobs\ProjectBackupJob\u0000daily\";b:1;}","commandName":"App\Jobs\ProjectBackupJob"},"job":"Illuminate\Queue\CallQueuedHandler@call","attempts":3}

What I do not understand is why Horizon does not keep failed job from a few hours ago on display and how to remedy that.

Second issue is clearly that I need better queueing so backup jobs of larger zipped files do work and do not clog up the queue. Rate limiting is now at 1500 requests per second at DO Spaces https://docs.digitalocean.com/products/spaces/details/limits/#rate-limits so that is double of what we had. So not sure why this 175MB zip file failed to be added. We do queue a lot of course, but why the MaxAttempts Exceeded Exception then.

Perhaps our handler

...
public function handle()
    {
        if ($this->project) {
            (new ProjectBackupService($this->project, $this->daily))->create();
            ProjectExport::dispatch($this->project);
        }
    }
...

needs an update or middleware added.

Questions

how to keep display of jobs in Horizon longer?
how to update handler to have larger files pass on to Digital Ocean Spaces better without failing?

Reading https://laravel.com/docs/9.x/queues#job-middleware some more on possible throttle and backoff options.. we use Redis as well as https://laracasts.com/discuss/channels/servers/laravel-digitalocean-spaces-rate-limits

rhand

3 years ago

Level 6

For question one on how to keep jobs run and failed jobs on display longer or to persist longer I found https://github.com/laravel/horizon/issues/275#issuecomment-433363871c in which Dries mentions

In your horizon.php config file there's a trim.recent setting which defaults to 60 minutes. You can increase this as you see fit to persists data longer. I'll send a PR to the docs to document this behavior.

We currently have

 /*
    |--------------------------------------------------------------------------
    | Job Trimming Times
    |--------------------------------------------------------------------------
    |
    | Here you can configure for how long (in minutes) you desire Horizon to
    | persist the recent and failed jobs. Typically, recent jobs are kept
    | for one hour while all failed jobs are stored for an entire week.
    |
    */

    'trim' => [
        'recent' => 60,
        'failed' => 10080,
    ],

so we can increase the recent trim time from 60 minutes to more. and for failed jobs too if need be.

For updating the job with some middleware for throttling and backoff time I am not quite done yet.

rhand

3 years ago

Level 6

As I use Redis for the queue QUEUE_DRIVER=redis I may need to update

...
'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => env('REDIS_QUEUE', 'default'),
    'retry_after' => 90,
    'block_for' => null,
    'after_commit' => false,
],
...

'redis' => [
    'driver' => 'redis',
    'connection' => 'default',
    'queue' => env('REDIS_QUEUE', 'default'),
    'timeout' => 180,
    'retry_after' => 200,
    'block_for' => null,
    'after_commit' => false,
],

and thereby add a longer timeout and retry_after. However on horizon.php I have

'environments' => [
    'production' => [
        'supervisor-1' => [
            'connection' => 'redis',
            'queue' => ['default', 'ssl-manager'],
            'balance' => 'auto',
            'minProcesses' => 1,
            'maxProcesses' => 15,
            'tries' => 3,
            'timeout' => 300,
        ],
    ],

so not sure which timeout will prevail then.

Please or to participate in this conversation.