Charrua
3 months ago
1152
1
Feedback

Queued jobs rate limiting without Redis

Posted 3 months ago by Charrua

Hello, on a previous discussion I was looking to use rate limiting when dispatching jobs to an external API. I know that the Laravel solution to this scenario is to use Redis, as the documentation exposes:

Redis::throttle('key')->allow(100)->every(60)->then(function () {
    // Job logic...
}, function () {
    // Could not obtain lock...

    return $this->release(10);
});

My app creates notifications to some users that are scheduled on some weekdays, each notification creates a job to send emails and SMS. So when the scheduler runs, it creates 3000 jobs (1500 for email and 1500 for SMS). With my actual setup, all jobs are dispatched immediately, so each API's will get those 1500 requests on the same minute, hitting rate limiting errors.

Looking for a solution using the database driver (as I do on my app) I have found one I wanted to share and see your thoughts.

Every time the scheduler runs, I query those 1500 members to notify and create chunks of 100, each chunk sends a notification with some delay. So when the worker starts to dispatch jobs it will find that they are not available to send all in the same minute, instead they will be available with the delay.

This is the code inside the handle method of the artisan command.

//query members to send notifications
$members = \App\Member::whereNull('expiration_date')->get();

$delay = 0;

//create chunks
$chunks = $members->chunk(100);

foreach ($chunks as $chunk){

    //send notification to chunk
    Notification::send($chunk, 
        (new AppIntroNewNoDate())->delay($delay));
            
    //add 60 seconds
    $delay += 60;

}

Please sign in or create an account to participate in this conversation.