Unexpected batch job processing order

Let's say I have four jobs:

DownloadMetadata
DownloadImage
ProcessImage
RespondToUser

The DownloadMetadata::dispatch() is triggered from a webhook handler. Within that job there are two other chained jobs that need to run:

class DownloadMetadata implements ShouldQueue {

		public function handle() {
		// logic and such

		foreach($images as $image) {
				Bus::chain([
						new DownloadImage(...),
			  			new ProcessImage(...),			
	])->dispatch();
							
	}}

The last job, ProcessImage triggers another job within itself (since it needs access to information that is not available at the time the chain is created, as it relies on outputs from the ProcessImage job):

class ProcessImage implements ShouldQueue {

		public function handle() {
		// logic and such

		RespondToUser::dispatch();
							
}}

My expectation is that when ran in queue worker, the result would look something like this:

DownloadMetadata
- DownloadImage
  - ProcessImage
    - RespondToUser
- DownloadImage
  - ProcessImage
    - RespondToUser
- DownloadImage
  - ProcessImage
    - RespondToUser

Where each "loop" of the last three jobs are for each $image. The indentation is just stylistic to highlight my understanding of the relationship between these.

However, regardless of how many queue workers I add, the result is the jobs are run incrementally by "type":

DownloadMetadata
- DownloadImage
- DownloadImage
- DownloadImage
- ProcessImage
- ProcessImage
- ProcessImage
- RespondToUser
- RespondToUser
- RespondToUser

...

While the end result is the same, it is less efficient than it could be, since I have to wait for all of the "Download" and "Process" jobs to complete before I tell the user anything, where instead I would expect to be able to process each $image incrementally and update the user along the way.

Any thoughts as to why the jobs would get picked up this way?

LaryAI

1 year ago

Level 58

The behavior you're experiencing is due to how the job dispatching and queue processing are structured. When you dispatch jobs in a chain, they are processed in the order they are added to the chain. However, if you want each image to be processed independently and notify the user as soon as each image is processed, you need to adjust the way you're dispatching the jobs.

Here's a solution to achieve the desired behavior:

Chain Jobs for Each Image Independently: Instead of chaining all DownloadImage jobs first, followed by all ProcessImage jobs, you should chain the DownloadImage, ProcessImage, and RespondToUser jobs for each image individually.
Dispatch Each Chain Separately: This ensures that each image is processed completely before moving on to the next image.

Here's how you can modify your code:

class DownloadMetadata implements ShouldQueue {

    public function handle() {
        // logic and such

        foreach($images as $image) {
            Bus::chain([
                new DownloadImage($image),
                new ProcessImage($image),
                new RespondToUser($image),
            ])->dispatch();
        }
    }
}

Explanation:

Bus::chain: This method is used to create a chain of jobs that should be processed in sequence. By creating a chain for each image, you ensure that the DownloadImage, ProcessImage, and RespondToUser jobs for each image are processed in order before moving on to the next image.
Independent Chains: Each image has its own independent chain of jobs. This means that as soon as the ProcessImage job is completed for an image, the RespondToUser job is dispatched immediately, allowing for incremental updates to the user.

By structuring your job dispatching in this way, you can achieve the desired processing order and improve the efficiency of your job processing.

bt3

1 year ago

Level 9

@LaryAI Sorry AI Lary, but your approach fails to take into account that I cannot trigger the RespondToUser job within the same batch as the information to process it is not available. And secondly, it stands to reason that your code sample is identical to mine less the RespondToUser job.

If implemented, I would expect the same result wherein all the DownloadImage jobs would process before the ProcessImage jobs.

My hunch is this is mainly an outcome of the foreach loop completing fast enough that all of the "batches", and thus the DownloadImage job is already queued for all images, before the first ProcessImage job is allowed to queue. So maybe the issue lies somewhere in how batch jobs are queued and started.

Snapey

1 year ago

Level 122

How many workers?

bt3

1 year ago

Level 9

@Snapey I've tested with 1 worker, as well as 4 workers. In both cases, all of the jobs still seemingly run by type instead of incrementally.

Snapey

1 year ago

Level 122

@bt3 I would do some experiments forcibly sleeping each job by some measurable time.

I would expect the download image to all be started together, then when the first image finishes, the first processimage is added to the queue (after all other jobs). What happens next is either the first RespondToUser is queued for the next free worker, or the second processImage is queued depending on what finishes first.

First step might be to not run any workers and check the queue contents after the chain has been dispatched.

Please or to participate in this conversation.