otepas's avatar
Level 13

Execute queued task by specific worker server

I need to perform the following task:

  • Download lots of files from cloud storage
  • Create ISO image
  • Upload resulting file to cloud storage

As we could have thousands of files, I thought of queuing the download of each file (one job per file). However, depending on the load, we could have more than one worker server. And obviously all files need to be downloaded by the same server in order to create the ISO file.

I thought of associating the jobs with the IP of the worker server. If that server picks a job, it is executed. If it is another server, the job is queued again.

Do you think it is a sensible approach? Can you think of another more elegant approach?

Thanks

0 likes
0 replies

Please or to participate in this conversation.