kill not really killing (Horizon)

This is just a copy of https://github.com/laravel/horizon/issues/666 but thought I would see if anyone has some input here.

Please see: https://github.com/laravel/horizon/blob/3.0/src/Console/PurgeCommand.php#L96

I have "laravel/framework": "5.8.*" with "laravel/horizon": "^3.2", running on:

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)

php -v
PHP 7.2.22 (cli) (built: Sep 11 2019 18:11:52) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

DB_CONNECTION=pgsql
QUEUE_CONNECTION=redis
REDIS_PREFIX="some_prefix:" # See: https://github.com/laravel/horizon/issues/279#issuecomment-473617301

with:

cat /etc/supervisord.d/app-worker.ini
[program:app-horizon]
process_name=%(program_name)s
command=php /var/www/sites/app/current/artisan horizon
autostart=true
autorestart=true
user=app
numprocs=1
redirect_stderr=true
stdout_logfile=/var/www/sites/app/current/storage/logs/worker.log

Running php artisan horizon:purge does not actually terminate the orphaned procs. Using kill -9 on the other hand does. Can we rough it up somehow? 🥊

Let me know if you need more details.

Also, I am having issues with horizon:terminate. Similar to https://github.com/laravel/horizon/issues/178#issuecomment-341272210

I run horizon:terminate and horizon:purge as part of an automated deploy but are left with a bunch of orphans.

If I try to run the commands interactively on the server I get the same result:

[root@host ~]# service supervisord start
Redirecting to /bin/systemctl start supervisord.service

[app@host current]$ ps aux |grep horizon | grep -v grep
app    5055  0.2  1.9 480308 36716 ?        S    13:24   0:00 php /var/www/sites/app/current/artisan horizon
app    5073  0.1  1.9 480308 36824 ?        S    13:24   0:00 /usr/bin/php artisan horizon:supervisor host-zvA7:supervisor-1 redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --balance=simple --max-processes=3 --min-processes=1 --nice=0
app    5088  0.1  1.9 478260 35880 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1
app    5089  0.1  1.9 478260 35876 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1
app    5090  0.1  1.9 478260 35872 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1

[app@host current]$ php artisan horizon:terminate
Sending TERM Signal To Process: 5055

[app@host current]$ ps aux |grep horizon | grep -v grep
app    5088  0.0  1.9 478260 35880 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1
app    5089  0.0  1.9 478260 35876 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1
app    5090  0.0  1.9 478260 35872 ?        S    13:24   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-zvA7:supervisor-1
app    5800  1.5  1.9 480308 36716 ?        S    13:28   0:00 php /var/www/sites/app/current/artisan horizon
app    5814  1.7  1.9 480308 36824 ?        S    13:28   0:00 /usr/bin/php artisan horizon:supervisor host-40Wa:supervisor-1 redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --balance=simple --max-processes=3 --min-processes=1 --nice=0
app    5828  1.7  1.9 478260 35872 ?        S    13:28   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-40Wa:supervisor-1
app    5829  1.7  1.9 478260 35876 ?        S    13:28   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-40Wa:supervisor-1
app    5830  1.7  1.9 478260 35880 ?        S    13:28   0:00 /usr/bin/php artisan horizon:work redis --delay=0 --memory=128 --queue=default --sleep=3 --timeout=60 --tries=3 --supervisor=host-40Wa:supervisor-1

[app@host current]$ php artisan horizon:purge
Observed Orphan: 5088
Observed Orphan: 5089
Observed Orphan: 5090

[app@host current]$ php artisan horizon:purge
Observed Orphan: 5088
Observed Orphan: 5089
Observed Orphan: 5090
Killing Process: 5088
Killing Process: 5089
Killing Process: 5090

Sometimes it claims to kill when running horizon:purge, but it does not actually terminate the proc.