Job timeouts whilst streaming large file from S3 to Cloudflare R2 despite increasing timeout to 40mins

Hi,

I'm attempting to stream PDF data from S3 to Cloudflare R2, using Laravel Filesystem along with league/flysystem-aws-s3-v3, this process is done via a queue. Most of the time it works perfectly but then very rarely there will be a failed transfer. On failure, I release it back to the job queue with the hope it will complete successfully next time.

The file link from S3 is revealed by an API. I've tried to debug as much as I can to see what might be causing the process to fail and narrowed it down to saving to R2 $disk->put(), is there a way I can get more information from this? It may not necessarily provide a fix but would good to be able to provide feedback back to client.

Thank you!

P.s. had to space out url scheme as the forum still thinks this is my first day/post, I've been here for a while now, does anyone else get this problem?

$apiResponse = Http::get('h t t p s://url.com?', [
								'key' => 'keystring',
								'fileId' => $orderProduct->file_name,
							]);

if($apiResponse->failed()){
		 if($apiResponse->clientError()){
				Log::error($apiResponse->clientError());
		}
		 if($apiResponse->serverError()){
				Log::error($apiResponse->serverError());
		 }
		Log::error('Released back to jobs and retry at '.now()->addMinutes(360));
		$this->release(now()->addMinutes(360));
}

if($apiResponse->successful()){
		Log::info('200 status range response for order');

		$fileLink = $apiResponse->body();
		$targetFile = $this->createFilename();

		$disk = Storage::disk('r2');
		$result = $disk->put(
                                        $targetFile,
                                        fopen($fileLink, 'r'));

		if($result == true){
				Log::info('Move to Cloudflare successful');
		}else{
				Log::error('Move to Cloudflare FAILED');
				Log::error('Released back to jobs and retry at '.now()->addMinutes(360));
				$this->release(now()->addMinutes(360));
		}
 }

UPDATE: I set throw to true in the filesystem definition and placed the download portion of the code in a try/catch but still no error returned. The job still fails after 3 attempts.

try{
	$disk = Storage::disk('r2');
    $result = $disk->put(
                     	$targetFile,
                        fopen($fileLink, 'r'));
} catch (Throwable $e){
	Log::error('Could not Upload file '.$e->getMessage());
}

So this has lead me to thinking about timeouts and whether the files being streamed and uploaded is particularly large. I examined the particular PDF and it's 1.6GB in size. I've looked at the failed_jobs table and can see in the exception that it does indeed timeout. So I increased timeouts by 10 minutes each time and the current time is 40 minutes (Supervisor also set to 40 minutes timeout)

//app/Jobs/MyJob.php
public $timeout = 2400;
//config/queue.php
 'retry_after' => 2400,

It's still failing! I'm about to increase it to an hour and see what happens, but surely 40mins should be enough?! The site is on it's own nix VM, there's nothing particularly taxing going on with the machine. Should I be approaching the stream/upload in a different way? I'm considering changing the code and actually downloading the file locally before uploading but streaming is more efficient on memory.

Is there a way when a job runs to see how many attempts it has made previously? For example if the job is being attempted for the 3rd time (3rd before failure) then I could insert a condition which downloads and uploads instead of streaming (I want to make streaming the default option as it works the majority of the time and only revert to traditional download/upload as last resort).