Is it good practice to have a cron-jobs in a docker swarm services?

I have dockerized my application, in the backend part I have hourly scheduled tasks that take 4 to 5 minutes, I used crontab to schedule the tasks, but the problem is when I update the service immediately the underlying container will be destroyed regardless of it is in the middle of the hourly process or not, and some times this will cause the task to be interrupted.

NOTES:

For docker-swarm management and update I use portainer.
The hourly task is downloading and processing and inserting some data into the DB.
We use CI/CD when a code is pushed to GitHub a new image will be created and tested and the portainer will take the new image and update the stack accordingly.

LaryAI

2 years ago

Level 58

One solution to this problem is to use a job queue system like RabbitMQ or Redis. Instead of running the task directly with crontab, you can add the task to the job queue and have a separate worker service that listens to the queue and processes the tasks. This way, when you update the service, the worker service will continue to process the remaining tasks without interruption.

Here's an example of how you can implement this using Redis and Laravel:

Install the Redis driver for Laravel:

composer require predis/predis

Add the Redis connection details to your .env file:

REDIS_HOST=redis
REDIS_PASSWORD=null
REDIS_PORT=6379

Create a new job class that implements the ShouldQueue interface:

php artisan make:job ProcessDataJob --queued

In the handle method of the job class, add the code to download, process, and insert the data into the database.
In your controller or command, add the following code to dispatch the job to the queue:

ProcessDataJob::dispatch();

Create a new worker service in your docker-compose file that runs the php artisan queue:work command:

worker:
  image: your-app-image
  command: php artisan queue:work --tries=3
  depends_on:
    - redis

Update your crontab to add a new task that runs the php artisan schedule:run command every minute:

* * * * * cd /path-to-your-project && php artisan schedule:run >> /dev/null 2>&1

This will ensure that the scheduled tasks are added to the queue every hour, and the worker service will process them without interruption even when the service is updated.

Please or to participate in this conversation.