In step two you can detect if the chunk is last. If it is last chunk then send boolean flag to that last job that it should fire new job which will then pick up where things were left off. Same applies for following jobs. Only dispatch them when the current is finished.
How To: Process multiple different jobs as chunks & chain them by type
Hey guys,
I'm currently trying to process several different jobs as chunks and then also chain them by type. Small overview of what I'm trying to accomplish:
- Artisan command downloads analytics csv file from a third-party api.
- Laravel Excel package processes csv file in chunks and queues each chunk as separate job, each job writes the chunk into a tmp database table.
- After all chunks from #2 are written to the tmp table, I need to process the tmp table and
updateOrCreate()each row in the aggregated reports table. Again, as chunks of jobs. - After all chunked jobs of #3 completed, I need to calculate specific metrics and write them to another database table, again as chunked jobs.
- After all jobs of #4 completed, I would need to export all of the calculated report rows, as chunks and write them to a csv file which is then uploaded back to that third-party API from step #1.
What currently throws me off is that I don't know how to make sure that each step (Job), is only triggered after all instances of the previous Job have completed.
I know there is a method withChain(), but I'm very unsure how to use that given that every Job will have many Instances running that all process only a chunk.
Any ideas? :)
PS: I think the ideal scenario would be if I could fire an event after all instances of each Job type have completed, to trigger the next chunked Job type. But totally unsure how to go about that
Please or to participate in this conversation.