Dec 12, 2016
0
Level 13
Execute queued task by specific worker server
I need to perform the following task:
- Download lots of files from cloud storage
- Create ISO image
- Upload resulting file to cloud storage
As we could have thousands of files, I thought of queuing the download of each file (one job per file). However, depending on the load, we could have more than one worker server. And obviously all files need to be downloaded by the same server in order to create the ISO file.
I thought of associating the jobs with the IP of the worker server. If that server picks a job, it is executed. If it is another server, the job is queued again.
Do you think it is a sensible approach? Can you think of another more elegant approach?
Thanks
Please or to participate in this conversation.