Jan 9, 2025
0
Level 3
Concurrent jobs throttling, limit is breached
Hey everyone. We need to limit the number of requests to an external API to 45 per second. So, I wrapped all the API calls into a queued job and implemented rate limiting there (throttling code is taken from the Laravel documentation). It doesn't work perfectly, the limit is frequently breached and we end up doing more than 45 requests per second. We do a lot of requests in parallel running multiple processes in Laravel Horizon. Jobs are being dispatched in batches of tens or hundreds a few times per minute. So, I suppose there's a race condition with the throttle counter somehow? Atomic lock failing? Can this be fixed?
We're running Laravel 8, code attached below. Thanks in advance.
public function handle()
{
Redis::throttle($this->throttleKey)
->block(0)
->allow(static::MAX_REQUESTS_PER_SECOND)
->every(1) // every 1 second
->then(function () {
// Lock obrained - handle job...
$this->doTheJob();
}, function () {
// Could not obtain lock...
$this->attemptsLeft--;
if ($this->attemptsLeft < 1) {
throw new \Illuminate\Contracts\Cache\LockTimeoutException('API request - lock timeout exceeded.');
}
usleep(1000 * 50); // 50ms
return $this->handle();
});
}
Please or to participate in this conversation.