Hi,
First of all I don't think that this is a Laravel issue, I'm really trying to figure out the problem of Str::random() generating repeated values - as it uses random_bytes().
Our scenario:
- 3 VPS servers running on Digital Ocean, 2 NYC1 and 1 NYC3 connected through VPN
- Ubuntu 18.04
- PHP 7.4 running with Laravel 7.12
- MySQL Galera Cluster 5.7.29
- Supervisor 3.3.1
- Redis Server 5.0.8 and Redis Sentinel 5.0.8
On each server, we have queues running using Supervisor with a config like this (modified to be posted here)
[group:my-app]
programs=my-app-queue-1,my-app-queue-2,my-app-queue-3
[program:my-app-queue-1]
process_name=%(program_name)s_%(process_num)02d
command=php7.4 /dir/app/artisan queue:work --queue=cron --sleep=3 --tries=2
directory=/dir/app
autostart=true
autorestart=true
startretries=10
numprocs=3
serverurl=AUTO
stopasgroup=true
stopsignal=TERM
killasgroup=true
stopwaitsecs=90
[program:my-app-queue-2]
process_name=%(program_name)s_%(process_num)02d
command=php7.4 /dir/app/artisan queue:work --queue=backup --sleep=3 --tries=2
directory=/dir/app
autostart=true
autorestart=true
startretries=10
numprocs=1
serverurl=AUTO
stopasgroup=true
stopsignal=TERM
killasgroup=true
stopwaitsecs=90
[program:my-app-queue-3]
process_name=%(program_name)s_%(process_num)02d
command=php7.4 /dir/app/artisan queue:work --sleep=3 --tries=2
directory=/dir/app
autostart=true
autorestart=true
startretries=10
numprocs=3
serverurl=AUTO
stopasgroup=true
stopsignal=TERM
killasgroup=true
stopwaitsecs=90
We have an API route /api/cron which receives a call from AWS Lambda every 1 minute and running the following code:
Artisan::queue('schedule:run')
->onQueue('cron');
return response()->json();
Our schedule method app/Console/Kernel.php is like this:
protected function schedule(Schedule $schedule) {
$crons = Cron::enabled()->get();
foreach ($crons as $cron) {
switch ($cron->cron_type) {
case CronType::ANSIBLE:
$schedule
->job(
new AnsibleCronJob($cron, Str::random(64)),
'cron'
)
->cron($cron->cron_expression);
break;
case CronType::COMNAND:
$schedule
->job(
new CommandCronJob($cron, Str::random(64)),
'cron'
)
->cron($cron->cron_expression);
break;
}
}
}
What is really strange is Str::random(64) generating repeated values and more than once, some values have repeated more than 100 times.
Things we have already made:
- Rebooted the servers
- Restarted the Supervisor service
- When we deploy, we run php artisan queue:restart, we also tried by restarting the Supervisor service
We don't thing it's a racing condition in this case, because we generate the value and only after we store it in the database, since Str::random() uses random_bytes, it should provide enough uniqueness for this situation. Str::uuid() falls into the same case, we were using it before.
Our guess is that something could be left in memory on the worker, but it's not clear.
We would really appreciate some ideas on the implementation / problem.
We are running things like this to avoid single point of failure (SPOF) and provide high availability and fault tolerance, as our previous scenario were running on a single server.