My app is processing a very large file, and to do the job I split the process part by part and push all of the parts into the queue.
There are multiple workers that will run these jobs.
The last part being processed will responsible to compile the whole output, summarise it, and cleanup the temporary tables and caches, to be used by next batch.
However, the user may queue another large file while the first batch is still being processed. Now, how to pause this second batch jobs until the first batch totally completed?
Does anyone has this similar case?
One simple idea would be some kind of mutex, e.g. in a database. So the first job would set a mutex and the last one unlocks it. Meanwhile, if another queue worker picks up the next batch, it won't start and can just sleep() a reasonable time. Probably not the the most elegant or Laravel-ish solution...
Do you want the user to be able to queue multiple large files?
You could always soft lock it from the UI until the first job is complete first
There's no way you can split it out and have both jobs running at the same time?
You could always have it use the Job's ID as a part of the temporary table's name to keep the temporary tables apart from each other.
If its a really large project you might want to containerize the entire job process in something like a docker container and have each job do it's job in a contained environment which means that the only thing you care about is the output and you won't have to worry about the jobs interfering with eachother
@repat Your idea made me think of a similar way that when a worker picks up the next batch, the first thing the job will do is to check for the previous batch, and wait until it's ready. But this time, it will push back itself into the queue if the prev batch is still running. But this sounds weird and wasting server resources since the worker always busy doing "nothing".
I wish there's a better way to solve this issue.
@webkenth I currently made it like that from the UI, locking other files while one is still in process. But then our client wants it possible to queue several files at once.
At first I think, well, it's easy...just dump all user's requests to queue, and process it one by one. But it turns out it's not that easy because it must combine async and sync process together in the queue. So I need to find another way to do this.
This is actually a small project and this app is actually just a helper to pre process large files before imported to their core system. So I think containers might be not fit in my case.
@rick20 Have a look at job chaining https://laravel.com/docs/7.x/queues#job-chaining maybe this can help you?
If you chain all your jobs they will be executed in sequence and in one fails the sequence stops. So for each batch you create a chain.
As far as the tmp tables and caches, I would assign to the batch a unique id and add that to the tables, so that each batch can clear what has done according to this id.
Please or to participate in this conversation.