Dynamically set timeout AND retryAfter

SO I have a bit of a weird situation (and certain things cannot be changed which I'll highlight). We have a Laravel 5.7 app with a single Job class. The job itself will fetch data and process it. The amount of data (and therefore the length of time the job takes) can vary massively so wanted to have a dynamic timeout based on the runtime of the past n jobs (the details are stored so can be queried). This is fine, I can dynamically work out the timeout value and set it as public $timeout = n as the job is posted to the queue.

The issue is that the retry_after value is set in the config/queue.php file and is read once so cannot be changed. We want all jobs to be run just once, so I've got --tries=1 which is great, but it means this scenario can happen as an example:

queue.php has timeout set to 30s
scheduler calculates timeout required for this specific job to be 50s and sets that as job is put in queue
job starts and passes 30s threshold
next worker sees this and sets it as failed

I don't get how to set a dynamic retry_after value. If timeout can be dynamic it doesn't seem to make sense that retry_after has to be hard set. I wanted to have it so that retry_after is timeout value plus a percentage over.

As I said about some things being locked, the time a job will run is totally dynamic, so having separate queues or a specific timeout isn't possible, it has to be calculated as its queued

It's also really infuriating that according to the divinglaravel posts he said:

retry_after in config/queue.php is same as --retry_after on command line
retryAfter as function or variable is NOT the same as either of the above, but is the same as --delay on command line
delay as function or variable is NOT the same as --delay on command line

sr57

4 years ago

Level 39

We want all jobs to be run just once, so I've got --tries=1

Once or one retry max ?

Assuming one retry

a single Job class

You can launch 2 times your job in the queue and test on the beginning that this job is already executed Ok to not run it 2 times;

PS I don't know if you can adjust the retry_after and use queue:restart, but even if this works it more complicated then the previous solution for a single job.

nikeMeal

4 years ago

Level 1

@sr57 so a job will be scheduled to run at (for example) 12pm. we want it to be attempted to run once. if that run fails or succeeds is sort of besides the point as it will be scheduled again at an interval.

the reason of running it as a job is so that only one instance of the job can ever run at any one time so if the 12pm job hangs, it would time out and run at 12:10 for example. but not as a retry, as a new job.

so when that job hits the timeout value, it is killed but wont ever have another one queued because it has to hit the retry_after value to try to run again, see its one attempt and then set as failed right?

all im really after is the ability to have a timeout value dynamically be set, and if the service does time out, its able to queue another instance of that job.

sr57

4 years ago

Level 39

Maybe you can define several queues with different retry_after and select the one to use versus the timeout you define per job.

but wont ever have another one queued because it has to hit the retry_after

should be "will be declared failed after hitting hit the retry_after

You should be able to launch a new job after an elapsed time equal to the job timeout (+ a delay you define of course)

jewelhuq

4 years ago

Level 18

Hello, Maybe relevant : https://laravel.com/docs/9.x/queues#max-job-attempts-and-timeout

public $timeout = 0; public $tries = 5; public function retryUntil() { return now()->addMinutes(10); }

Please or to participate in this conversation.