how to conditionally chain queue jobs

I am using laravel-excel to export data to excel, it can take quiet a while to finish and sometimes timeout when the data is very large. I decided to break the export into smaller units and combine them later. The user needs to be notified when the whole process is complete. the problem is that the user gets notified upon every completed piece of the process. for instance if i break the export process into five pieces the user will be notified five times and that's not what i want, so i created another job and chained the user notification on but the problem with this is that the user get notified before the exports get completed.

Is There a way i can conditionally chain a job so they can run sequentially.

note am using queues for the process

  $pathToFile = Str::slug(auth()->user()->name).'/'.md5(collect(request()->query())->flatten());
            $masterExcelFileName = Str::slug(request()->user()->name.' '.md5(collect(request()->query())->flatten())).'.xlsx';
            $this->dispatch(
                (new  ProcessDownload($this->getGroup(), $this->getSubGroup(), $this->getDateFrom(),
                    $this->getDateTo(), request('system_name'), $pathToFile, $masterExcelFileName)
                )->chain([
                    new NotifyUserOfCompletedExport(request()->user(), $pathToFile, $masterExcelFileName),
                ])->allOnQueue('exports')
            );

ProcessDownload class

 public function handle()
        {
// only notify user when this whole process is completed

            collect($this->datePeriod())->chunk(7)->each(function (Collection $week) {
                $days = $week->map(function ($day) {
                    return $this->dayTimeRange($day);
                });
                $filename = Str::slug(sprintf('%s %s %s %s %s',
                        $week->first()->format('M j Y'),
                        'to',
                        $week->last()->format('M j Y'),
                        $this->group,
                        $this->subGroup)).'.xlsx';

                (new SmpWorkbookExport(
                    $this->group,
                    $this->subGroup,
                    $this->name,
                    $days
                ))->queue("$this->pathToFile/$filename", 'public', Excel::XLSX)
                    ->chain([
                        new  MergeWorkSheets($this->pathToFile,$filename,$this->masterExcelFileName)
                    ])
                    ->allOnQueue('exports');
            });

Braunson

7 years ago

Level 18

You can use the normal job chaining..

ProcessDownload::withChain([
    new SmpWorkbookExport($data),
    new MergeWorkSheets($data)
])->dispatch($data);

Pass an array of data needed for each job so you can do the logic calculation inside each job. If a chained job needs not to run you can do the check inside the job and either $job->delete() or return false to end the job.

If you need to pass data from one chained job to another sequentially, I don't have a solid Laravel answer but Guzzle's Promises may be an option.. check this https://stackoverflow.com/a/46801397/610880

Additionally if you need to pass data to another job in the chain, you could use something like redis/cache to temporarily store the data to pass it down the line and at the end delete/empty that cached data (cached data is set in the first queue job, cleared out in the last when it's no longer needed)

Sabonzy

7 years ago

Level 9

@BRAUNSON - Thanks for your response. Inside ProcessDownload job multiple instance of SmpWorkbook job will run depending on the chunk size, queuing the job before check whether the job should be run might end up filling the queue with unnecessary jobs and i need all instances of SmpWorkbook to finish before any related job like MergeWorkSheets($data) or NotifyUserOfCompletedExport($data) can run

Sabonzy

7 years ago

Level 9

Current am firing an event and conditionally pushing event listeners to queue as describe here by Mohamed.

event(new ExportCompletedEvent($this->user, $pathToFile, $fileToMergeInto,$this->shouldNotifyUser($days))
);

and in the Event Listener check whether it should be queued or not

 public function shouldQueue($event)
        {
            return $event->shouldNotifyUser;
        }

even though it doesn't solve the problem it reduces the time difference between when the user is notified and the export been ready

Please or to participate in this conversation.