Feb 11, 2026
0
Level 1
We migrated self hosted redis to Valkey 8.1 serverless aws elasticache.
Hi All,
Old
- AWS EC2 for application servers
- AWS EC2 for horizon server
- AWS EC2 for self hosted Redis
New
- AWS EC2 for application servers with ASG
- AWS EC2 for horizon server
- Valkey Serverless Cluster Mode
Recently we moved self hosted redis 7.2 to valkey 8.1 culster mode. We update all configuration for slot and all. but as we are running bulk cmapaign before migration the horizon was sending all jobs perfectly. But after migration in between many jobs just missed out.
Example
We running a campaign where audience is 4000, once campaign completed we get some of audience get the communication notification and messages.
Before Migration
Send all message to 4000 Audience
After migration
Sometime send to some people (random number) and sometime staying to 0.
No Failure message or jobs show its just skipped or not register silently. there is no code update for this campaigning.
current horizon configuration
<?php
use Illuminate\Support\Str;
return [
/*
|--------------------------------------------------------------------------
| Horizon Domain
|--------------------------------------------------------------------------
|
| This is the subdomain where Horizon will be accessible from. If this
| setting is null, Horizon will reside under the same domain as the
| application. Otherwise, this value will serve as the subdomain.
|
*/
'domain' => env('HORIZON_DOMAIN'),
/*
|--------------------------------------------------------------------------
| Horizon Path
|--------------------------------------------------------------------------
|
| This is the URI path where Horizon will be accessible from. Feel free
| to change this path to anything you like. Note that the URI will not
| affect the paths of its internal API that aren't exposed to users.
|
*/
'path' => env('HORIZON_PATH', 'horizon'),
/*
|--------------------------------------------------------------------------
| Horizon Redis Connection
|--------------------------------------------------------------------------
|
| This is the name of the Redis connection where Horizon will store the
| meta information required for it to function. It includes the list
| of supervisors, failed jobs, job metrics, and other information.
|
*/
'use' => 'default',
/*
|--------------------------------------------------------------------------
| Horizon Redis Prefix
|--------------------------------------------------------------------------
|
| This prefix will be used when storing all Horizon data in Redis. You
| may modify the prefix when you are running multiple installations
| of Horizon on the same server so that they don't have problems.
|
| IMPORTANT: For Valkey/Redis cluster mode, we use a hash tag {horizon}
| to ensure all Horizon metadata keys hash to the same slot, avoiding
| cross-slot errors during atomic operations.
|
*/
'prefix' => env(
'HORIZON_PREFIX',
Str::slug(env('APP_NAME', 'laravel'), '_').'{horizon}:'
),
/*
|--------------------------------------------------------------------------
| Horizon Route Middleware
|--------------------------------------------------------------------------
|
| These middleware will get attached onto each Horizon route, giving you
| the chance to add your own middleware to this list or change any of
| the existing middleware. Or, you can simply stick with this list.
|
*/
'middleware' => ['web'],
/*
|--------------------------------------------------------------------------
| Queue Wait Time Thresholds
|--------------------------------------------------------------------------
|
| This option allows you to configure when the LongWaitDetected event
| will be fired. Every connection / queue combination may have its
| own, unique threshold (in seconds) before this event is fired.
|
| NOTE: Queue names use hash tags {queue_name} for cluster mode compatibility.
|
*/
'waits' => [
'redis:{default}' => 60,
'redis:{otps}' => 10,
'redis:{campaignEmails}' => 120,
'redis:{campaignWhatsApps}' => 120,
],
/*
|--------------------------------------------------------------------------
| Job Trimming Times
|--------------------------------------------------------------------------
|
| Here you can configure for how long (in minutes) you desire Horizon to
| persist the recent and failed jobs. Typically, recent jobs are kept
| for one hour while all failed jobs are stored for an entire week.
|
*/
'trim' => [
'recent' => 60, // Keep as-is (short-lived)
'pending' => 1440, // safe for bulk campaigns
'completed' => 1440, // Optional: longer history
'recent_failed' => 10080, // 1 week
'failed' => 10080,
'monitored' => 10080,
],
/*
|--------------------------------------------------------------------------
| Silenced Jobs
|--------------------------------------------------------------------------
|
| Silencing a job will instruct Horizon to not place the job in the list
| of completed jobs within the Horizon dashboard. This setting may be
| used to fully remove any noisy jobs from the completed jobs list.
|
*/
'silenced' => [
// App\Jobs\ExampleJob::class,
],
/*
|--------------------------------------------------------------------------
| Metrics
|--------------------------------------------------------------------------
|
| Here you can configure how many snapshots should be kept to display in
| the metrics graph. This will get used in combination with Horizon's
| `horizon:snapshot` schedule to define how long to retain metrics.
|
*/
'metrics' => [
'trim_snapshots' => [
'job' => 24,
'queue' => 24,
],
],
/*
|--------------------------------------------------------------------------
| Fast Termination
|--------------------------------------------------------------------------
|
| When this option is enabled, Horizon's "terminate" command will not
| wait on all of the workers to terminate unless the --wait option
| is provided. Fast termination can shorten deployment delay by
| allowing a new instance of Horizon to start while the last
| instance will continue to terminate each of its workers.
|
*/
'fast_termination' => true,
/*
|--------------------------------------------------------------------------
| Memory Limit (MB)
|--------------------------------------------------------------------------
|
| This value describes the maximum amount of memory the Horizon master
| supervisor may consume before it is terminated and restarted. For
| configuring these limits on your workers, see the next section.
|
*/
'memory_limit' => 256,
/*
|--------------------------------------------------------------------------
| Queue Worker Configuration
|--------------------------------------------------------------------------
|
| Here you may define the queue worker settings used by your application
| in all environments. These supervisors and settings handle all your
| queued jobs and will be provisioned by Horizon during deployment.
|
| Defaults are merged with environment-specific configs. Using minProcesses=1
| ensures gradual startup. balanceCooldown prevents rapid scaling.
|
| IMPORTANT: All queue names use hash tags {queue_name} to ensure all
| related Redis keys (queue list, reserved, delayed, etc.) hash to the
| same slot in Valkey/Redis cluster mode. This prevents CROSSSLOT errors
| and ensures atomic operations work correctly across shards.
|
*/
'defaults' => [
'main-supervisor' => [
'connection' => 'redis',
'queue' => ['{default}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 512,
'tries' => 1,
'timeout' => 7200,
'nice' => 0,
],
'trigger-add-supervisor' => [
'connection' => 'redis',
'queue' => ['{trigger-add}'],
'balance' => 'auto',
'autoScalingStrategy' => 'time',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 256,
'tries' => 1,
'timeout' => 7200,
'nice' => 0,
],
'campaign-sendnow-supervisor' => [
'connection' => 'redis',
'queue' => ['{SendCampaignMessages}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 5,
'maxProcesses' => 25,
'balanceMaxShift' => 3,
'balanceCooldown' => 2,
'memory' => 512,
'tries' => 1,
'timeout' => 7200,
'nice' => 0,
],
'campaign-email-supervisor' => [
'connection' => 'redis',
'queue' => ['{campaignEmails}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 50,
'balanceMaxShift' => 3,
'balanceCooldown' => 5,
'memory' => 512,
'tries' => 1,
'timeout' => 7200,
'nice' => 0,
],
'campaign-whatsapp-supervisor' => [
'connection' => 'redis',
'queue' => ['{campaignWhatsApps}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 512,
'tries' => 1,
'timeout' => 7200,
'nice' => 0,
],
'otps-supervisor' => [
'connection' => 'redis',
'queue' => ['{otps}'],
'balance' => 'auto',
'autoScalingStrategy' => 'time',
'minProcesses' => 1,
'maxProcesses' => 1,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 128,
'tries' => 1,
'timeout' => 60,
'nice' => 0,
],
'leads-supervisor' => [
'connection' => 'redis',
'queue' => ['{lead_add}'],
'balance' => 'auto',
'autoScalingStrategy' => 'time',
'minProcesses' => 1,
'maxProcesses' => 1,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 128,
'tries' => 1,
'timeout' => 60,
'nice' => 0,
],
'logs-supervisor' => [
'connection' => 'redis',
'queue' => ['{logs}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 128,
'tries' => 1,
'timeout' => 60,
'nice' => 0,
],
'sms-supervisor' => [
'connection' => 'redis',
'queue' => ['{bulk_sms}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 128,
'tries' => 1,
'timeout' => 60,
'nice' => 0,
],
'intense-jobs-supervisor' => [
'connection' => 'redis',
'queue' => ['{intense_jobs}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 20,
'timeout' => 7200,
],
'analytics-supervisor' => [
'connection' => 'redis',
'queue' => ['{analytics}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 1,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 512,
'tries' => 1,
'timeout' => 60,
'nice' => 0,
],
'batch-supervisor' => [
'connection' => 'redis',
'queue' => ['{batch}'],
'balance' => 'auto',
'autoScalingStrategy' => 'size',
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'memory' => 256,
'tries' => 1,
'timeout' => 3600,
'nice' => 0,
],
],
'environments' => [
'testing' => [
'main-supervisor' => [
'maxProcesses' => 1,
],
'trigger-add-supervisor' => [
'maxProcesses' => 1,
],
'campaign-sendnow-supervisor' => [
'maxProcesses' => 1,
],
'campaign-email-supervisor' => [
'maxProcesses' => 1,
],
'campaign-whatsapp-supervisor' => [
'maxProcesses' => 1,
],
'otps-supervisor' => [
'maxProcesses' => 1,
],
'leads-supervisor' => [
'maxProcesses' => 1,
],
'logs-supervisor' => [
'maxProcesses' => 1,
],
'sms-supervisor' => [
'maxProcesses' => 1,
],
'intense-jobs-supervisor' => [
'maxProcesses' => 1,
],
'analytics-supervisor' => [
'maxProcesses' => 1,
],
'batch-supervisor' => [
'maxProcesses' => 1,
],
],
'staging' => [
'main-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 3,
],
'trigger-add-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
],
'campaign-sendnow-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
],
'campaign-email-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
],
'campaign-whatsapp-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
],
'otps-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 1,
],
'leads-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 1,
],
'logs-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 1,
],
'sms-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
],
'intense-jobs-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 3,
],
'analytics-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 1,
],
'batch-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
],
],
/*
|--------------------------------------------------------------------------
| Production Environment (Auto-Scaling Optimized)
|--------------------------------------------------------------------------
|
| - minProcesses=1 ensures minimal startup footprint
| - balanceMaxShift controls how many workers are added/removed per cycle
| - balanceCooldown (seconds) prevents rapid scaling oscillation
| - Workers scale up gradually based on queue size/wait time
|
*/
'production' => [
'main-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 60,
'balanceMaxShift' => 2,
'balanceCooldown' => 5,
],
'trigger-add-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 2,
'balanceCooldown' => 5,
],
'campaign-sendnow-supervisor' => [
'minProcesses' => 5,
'maxProcesses' => 25,
'balanceMaxShift' => 3,
'balanceCooldown' => 2,
],
'campaign-email-supervisor' => [
'minProcesses' => 2,
'maxProcesses' => 40,
'balanceMaxShift' => 3,
'balanceCooldown' => 5,
],
'campaign-whatsapp-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 15,
'balanceMaxShift' => 2,
'balanceCooldown' => 5,
],
'otps-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'leads-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'logs-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'sms-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'intense-jobs-supervisor' => [
'minProcesses' => 5,
'maxProcesses' => 20,
'balanceMaxShift' => 2,
'balanceCooldown' => 5,
],
'analytics-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'batch-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
],
/*
|--------------------------------------------------------------------------
| Fallback for Other Environments
|--------------------------------------------------------------------------
*/
'*' => [
'main-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 5,
],
'trigger-add-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 5,
],
'campaign-sendnow-supervisor' => [
'minProcesses' => 5,
'maxProcesses' => 25,
'balanceMaxShift' => 3,
'balanceCooldown' => 2,
],
'campaign-email-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 2,
'balanceCooldown' => 5,
],
'campaign-whatsapp-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 5,
],
'otps-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'leads-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'logs-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 2,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'sms-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 3,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'intense-jobs-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 5,
'balanceMaxShift' => 1,
'balanceCooldown' => 5,
],
'analytics-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 1,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
'batch-supervisor' => [
'minProcesses' => 1,
'maxProcesses' => 3,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
],
],
],
];
Please or to participate in this conversation.