Stop supervisor (not queue workers) when queues are empty

Hi, i have some long running queues correctly managed by different supervisor programs.

After some hours the queues are empty, but supervisor still launches workers, even if there's nothing to do.

I'd like to stop supervisor when a queue is empty: i know that the workers restarted by supervisor are short-lived when the queues are empty, but in my particular situation they're a lot of parallel short-lived procs, affecting server performance.

Queue works command are issued with the --stop-when-empty option, here down a summary:

[program:backups-line-1-worker]
process_name=%(program_name)s_%(process_num)02d
command=/opt/plesk/php/8.0/bin/php /path/to/artisan queue:work --queue=backups_line_1 --stop-when-empty
numprocs=5
autostart=false
autorestart=true
stopasgroup=true

[program:backups-line-2-worker]
process_name=%(program_name)s_%(process_num)02d
command=/opt/plesk/php/8.0/bin/php /path/to/artisan queue:work --queue=backups_line_2 --stop-when-empty
numprocs=3
autostart=false
autorestart=true
stopasgroup=true

Quite obviously, i'm launching them (with all Jobs Batches already dispatched) with

supervisorctl start backups-line-1-worker:*
supervisorctl start backups-line-2-worker:*

Now:

i guess that the --stop-when-empty options refers to the queue work command; not affecting supervisor behaviour;
i don't like to issue a command (e.g. supervisorctl stop backups-line-2-worker:*) via crontab to stop supervisor after X hours, i'd like to stop supervisor right after specific queue is empty;
i'd prefer not to mess with exec('supervisorctl stop backups-line-2-worker:*') as a last job of each laravel batch->dispatch()

Is there any elegant solution? :)

rodrigo.pedra

4 years ago

Level 56

i guess that the --stop-when-empty options refers to the queue work command; not affecting supervisor behaviour;

Yes.

If you want to stop supervisor when the queue is empty, how would you automatically start it again when there are new jobs in the queue? Doesn't manually stopping and starting supervisor defeats its purpose?

If your issue is supervisor restarting processes two quickly when your queues are a long time empty, consider using the -sleep option instead of the --stop-when-empty option.

As stated in the docs, the --stop-when-empty options is useful when you want to process the queue and be done:

The --stop-when-empty option may be used to instruct the worker to process all jobs and then exit gracefully. This option can be useful when processing Laravel queues within a Docker container if you wish to shutdown the container after the queue is empty:

reference: https://laravel.com/docs/9.x/queues#processing-all-queued-jobs-then-exiting

Something like this:

command=/opt/plesk/php/8.0/bin/php /path/to/artisan queue:work --queue=backups_line_1 --sleep=60

This will make the queue sleep for 60 seconds when the queue is empty, without killing the process.

I guess what is impacting your server's performance is the creation/killing of many process in a a short time. An idle queue worker should not have a noticeable impact on your server's memory nor performance.

To avoid memory leaks, I usually combine my workers with --sleep (with a lower --sleep value) and --max-jobs. So the worker is not checking the queue frequently when it is empty (from the --sleep option), and its process gets killed once in a while to avoid eventual memory leaks from a very long running process ( from the --max-jobs options) as supervisor will restart the worker when it is killed.

Snapey

4 years ago

Level 122

your queue workers should run all the time, and supervisor is there to ensure those workers keep running

You seem to be overthinking it?

if you really want dynamic provisioning of workers then install Horizon

Please or to participate in this conversation.