I have multiple EC2 instances running Horizon workers that are monitoring the same queues. I want to automatically scale the number of EC2 instances up and down based on time of day.
I have all of this working - multiple instances running, and I can scale up/down using AWS's auto-scaling groups. What I haven't yet figured out is a way to gracefully shut down Horizon before terminating an EC2 instance when scaling down. Some of my queued jobs can take a while to run and if one is running when AWS scales down, the EC2 instance terminates before that job finishes.
What's the best way to handle this?
Should I initiate scaling via Laravel's scheduler and hook into the AWS API? This would presumably let me terminate horizon before telling the AWS API to scale down (but can I even terminate horizon via code?).
Is there a way in AWS to issue server commands before scaling down? Could I call "php artisan horizon:terminate" from AWS as part of the scale down routine? If so, how does AWS know when in process jobs are finished?
Any other ideas?
Thanks!