Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

madsem's avatar

How To: Process multiple different jobs as chunks & chain them by type

Hey guys,

I'm currently trying to process several different jobs as chunks and then also chain them by type. Small overview of what I'm trying to accomplish:

  1. Artisan command downloads analytics csv file from a third-party api.
  2. Laravel Excel package processes csv file in chunks and queues each chunk as separate job, each job writes the chunk into a tmp database table.
  3. After all chunks from #2 are written to the tmp table, I need to process the tmp table and updateOrCreate() each row in the aggregated reports table. Again, as chunks of jobs.
  4. After all chunked jobs of #3 completed, I need to calculate specific metrics and write them to another database table, again as chunked jobs.
  5. After all jobs of #4 completed, I would need to export all of the calculated report rows, as chunks and write them to a csv file which is then uploaded back to that third-party API from step #1.

What currently throws me off is that I don't know how to make sure that each step (Job), is only triggered after all instances of the previous Job have completed.

I know there is a method withChain(), but I'm very unsure how to use that given that every Job will have many Instances running that all process only a chunk.

Any ideas? :)

PS: I think the ideal scenario would be if I could fire an event after all instances of each Job type have completed, to trigger the next chunked Job type. But totally unsure how to go about that

0 likes
7 replies
bugsysha's avatar

In step two you can detect if the chunk is last. If it is last chunk then send boolean flag to that last job that it should fire new job which will then pick up where things were left off. Same applies for following jobs. Only dispatch them when the current is finished.

madsem's avatar

Thanks @bugsysha meanwhile I came up with this:

DB::table('temporary_platform_reports')
          ->where('platform', $event->platformName)
          ->whereNull('processed')
          ->chunkById(100, function ($stats) {
              array_push($this->importJobs, new ImportPlatformReports($stats));
          });

        // grab first job from array to use in dispatch
        $firstJob = array_shift($this->importJobs);

        // dispatch import job chain
        dispatch($firstJob)->chain([
            $this->importJobs,
            new FirePlatformReportImportCompletedEvent,
        ]);
<?php

namespace App\Jobs;

use App\Events\PlatformReportImportCompleted;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class FirePlatformReportImportCompletedEvent
{

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct()
    {
        //
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        event(new PlatformReportImportCompleted);
    }
}

This seems to be doing what I wanted, basically I simply chain the first job type, then at the end execute a simple non-queueable job that fires an event and triggers the next listener and so forth.

You think this should work well?

bugsysha's avatar

Isn't chain dispatching events that you provide when the current/previous is successful? If so then FirePlatformReportImportCompletedEvent should be dispatched as many times as you have successful jobs.

madsem's avatar

No, the chain executes in order and only if all previous jobs completed successfully. So the Job which triggers an event is only executed once.

So far it seems to do exactly what I wanted

bugsysha's avatar

OK. Then there is no reason not to push all jobs to chain straight away. Why do you need a job which fires an event which then listeners picks up to dispatch more jobs?

madsem's avatar

Because all of the 5 steps are depending on each other, and each step needs the data created in the previous step. So I cannot simply throw it all in the queue but have to make sure the required data exists. Does that makes sense? :)

bugsysha's avatar

Yeah it does make sense. Just seems like a unnecessary step. But on top of my mind I can not think of any other approach which might be cleaner. Maybe I had too much for lunch and now there is no blood reaching my brain.

Please or to participate in this conversation.