Concurrent Jobs and Sequential Jobs and the Pipeline

I am building a download that has several parts to it. I have two tasks that pushes multiple jobs to the queue and those jobs can be run concurrently but need to finish before the next pipe.

$pipes = [
    AddTokenToDownloadField::class,
    AssembleDownload::class,
    MoveDownload::class,
    RemoveTokenFromDownloadField::class,
    SendEmailDownloadReady::class,
];

Can all of the jobs be queueable?

My current solution is to have a master job that is queable and then call each part above as a synchronous call. It works but I am trying to see if I can speed it up by pushing concurrent jobs on the queue and then waiting for them to complete before moving to the next pipe.

MarkP

6 years ago

Best Answer

Level 6

You cannot have concurrent jobs (in parallel) and keep everything working. Unless you implement a job monitoring system you cannot guarantee a job finishing before the next pipe is called. Instead I solved this in two ways. My first was was to an asynchronous job to call synchronous jobs (jobs without the implements ShouldQueue).

PrepareDownload::dispatch($payload);
AssembleDownload::dispatch($payload);
MoveDownload::dispatch($payload);
CleanUp::dispatch($payload);

However, I realized this was really a pipeline so my next was to use an asynchronous job to call a pipeline. I preferred this method because I could pass along a payload to each pipe and the pipe could build on that payload.

$pipeline = app(Pipeline::class);

$pipeline->send($payload)
    ->through([
      PrepareDownload::class,
      AssembleDownload::class,
      MoveDownload::class,
      CleanUp::class,
    ])->then(function ($payload) {
        Mail::to($user)->send(new SendUserFile($payload));
    });

1 like

Please or to participate in this conversation.