Charrua's avatar

Queued jobs rate limiting without Redis

Hello, on a previous discussion I was looking to use rate limiting when dispatching jobs to an external API. I know that the Laravel solution to this scenario is to use Redis, as the documentation exposes:

Redis::throttle('key')->allow(100)->every(60)->then(function () {
    // Job logic...
}, function () {
    // Could not obtain lock...

    return $this->release(10);
});

My app creates notifications to some users that are scheduled on some weekdays, each notification creates a job to send emails and SMS. So when the scheduler runs, it creates 3000 jobs (1500 for email and 1500 for SMS). With my actual setup, all jobs are dispatched immediately, so each API's will get those 1500 requests on the same minute, hitting rate limiting errors.

Looking for a solution using the database driver (as I do on my app) I have found one I wanted to share and see your thoughts.

Every time the scheduler runs, I query those 1500 members to notify and create chunks of 100, each chunk sends a notification with some delay. So when the worker starts to dispatch jobs it will find that they are not available to send all in the same minute, instead they will be available with the delay.

This is the code inside the handle method of the artisan command.

//query members to send notifications
$members = \App\Member::whereNull('expiration_date')->get();

$delay = 0;

//create chunks
$chunks = $members->chunk(100);

foreach ($chunks as $chunk){

    //send notification to chunk
    Notification::send($chunk, 
        (new AppIntroNewNoDate())->delay($delay));
            
    //add 60 seconds
    $delay += 60;

}
0 likes
1 reply
bobbybouwmann's avatar

This is perfectly fine! Third-parties with rate limiting can be a tough challenge.

Another approach would be having one job to fetch all 1500 users and dispatch a job with a delay per user or per chunk. Then the job itself can even delay the notifications if needed.

If your approach doesn't hit the rate limits at this point your good to go. Looks solid!

Please or to participate in this conversation.