Make a Job wait on several other Jobs

In general I would use the chaining functionality here, which will then look like this

"A" > "A-finished" > "B", "B", "B-finished", "C"

So the finished job would kick off all the other jobs. This way you can use a chain and you can use the last job to perform all the actions.

An alternative way is using a cronjob that checks if all A actions have been performed for a certain model/object and then pull all B tasks on the queue. Same goes for C that double checks if everything from B has run.

Both approaches are fine but it depends on your needs. Do you need them to run fastly after each other, I would give for the chains. Otherwise using a cronjob is a good alternative ;)

@bobbybouwmann

I'm not certain your first solution would work. Let's say the "A-finished" job gets dispatched, and then "B", "B", and "B-finished" are dispatched concurrently. If I had three queue listeners, then all three jobs would get picked up. This would cause "B-finished" to run early.

For you alternative solution, what would trigger the cronjob? My goal here is to essentially have both multi-threading (by running jobs in parallel) and process-dependency (by using chaining or some other approach).

My application currently has 7 processes that need to be chained, and each process currently runs actions in a loop over a series of models. However, each model-specific action is isolated to that model, so I'd like to run that part concurrently.

Here's the sort of workflow I'm expecting:

1. Dispatch the first process ("A")
2. "A" dispatches 5+ "A-record" processes
3. Something ("A", "A-record", or other) dispatches the second process ("B") when all "A-record" processes are complete
4. "B" dispatches 300+ "B-record" processes
5. Something dispatches the third process ("C"), similar to #3
6. "C" dispatches 8000+ "C-record" processes
7. etc.

Since my plan here is to use Laravel Vapor in the long run, when something like #6 happens, I'm expecting 8000+ queue listeners to spin up, handle the job, and spin down. I'm currently just operating in a basic Laravel environment, so I'll likely only have 1 to 4 listeners always running until I get this process ironed out. If I were to chain the 8000 job instances together, then they'd run sequentially, rather than in parallel.

I think my biggest problem here is knowing when to dispatch the next process that has a dependency (i.e. #3 and #5). I can envision various ways of tackling this problem, but they all of pros and cons that make me worry that I'm not thinking about the solution in the correct manner (hence why I'm coming here for insight).

@tylernathanreed

I know this thread is quite old, but I'm had a similar issue in one of my projects and solved it the following way:

What I'm doing is that my Job "A" dispatches all "B" jobs and saves whether they have been dispatched in a database table (using some unique key that the job uses to mark it as finished or failed). After "A" dispatches all Jobs "B" it dispatches itself again with a wait-time (1 hour for me, but could be related to the number of "B" Jobs). When it runs next it checks if all Jobs "B" are finished and dispatches failed "B" jobs again (or fails itself if one "B" job fails to often).

Once the "A" Job has established that all "B" Jobs have successfully finished it dispatches Job "C".

I'm not using the "job chaining" functionality of Laravel (not sure if it would handle the re-dispatching of the "A" Job)

Hope that my solution helps somewhat if this issue is still relevant.

@tylernathanreed

I'll add my "I know this thread is quite old" response 😉

You would likely be interested in the Venture package. It does exactly what you're looking for. The docs are here: https://laravel-venture.netlify.app/

This is the example from that page:

class PublishNewPodcastWorkflow extends AbstractWorkflow
{
    private Podcast $podcast;

    public function __construct(Podcast $podcast)
    {
        $this->podcast = $podcast;
    }

    public function definition(): WorkflowDefinition
    {
        Workflow::define('Publish new podcast')
            ->addJob(new ProcessPodcast($this->podcast))
            ->addJob(new OptimizePodcast($this->podcast))
            ->addJob(new ReleaseOnTransistorFM($this->podcast), [
                // These are the job's dependencies. The job will only
                // run when all of its dependencies have finished.
                ProcessPodcast::class,
                OptimizePodcast::class
            ])
            ->addJob(new ReleaseOnApplePodcasts($this->podcast), [
                ProcessPodcast::class,
                OptimizePodcast::class
            ])
            ->addJob(new CreateAudioTranscription($this->podcast), [
                ProcessPodcast::class,
            ])
            ->addJob(new TranslateAudioTranscription($this->podcast), [
                CreateAudioTranscription::class,
            ])
            ->addJob(new NotifySubscribers($this->podcast), [
                ReleaseOnTransistorFM::class,
                ReleaseOnApplePodcasts::class,
            ])
            ->addJob(new SendTweetAboutNewPodcast($this->podcast), [
                TranslateAudioTranscription::class,
                ReleaseOnTransistorFM::class,
                ReleaseOnApplePodcasts::class,
            ]);
    }
}

This creates this workflow:

You then run the workflow:

PublishNewPodcastWorkflow::start($podcast);

And as it says on the docs...

Venture will take care of running the jobs in the correct order, parallelizing jobs that don't have any interdependencies and waiting for all dependencies of a job to be resolved before starting it. It also provides a way to inspect a workflow and each of its jobs.