Level 8
You probably want to queue this.
https://laravel.com/docs/8.x/notifications#queueing-notifications
You can use a simple Redis Queue with Laravel Horizon.
Apr 8, 2021
15
Level 2
Notifications at scale > 1 million users
Does anyone have any tips or insights into how to send a notification (not mail) to a database of users with over a million records and counting.
Laravel docs indicate using:
Notification::send($users, new InvoicePaid($invoice));
This times out or runs out of memory. Is there a better way or does the infrastructure just need updating to handle loading so many records?
Thanks
Level 2
Yes it would be via a Redis queue, still times out or expires memory in generating over a million jobs going into the queue
Level 8
You can try to use a PHP generator for this. But maybe be careful with them. Jeffrey does a cool video on generators here on Laracasts
Level 2
Thanks will take a look and see if generators help in this instance
Level 73
You can also use chunking so that you send maybe 4000 every itteration.
Level 2
Tried that, it never gets to complete with either small chunks or large chunks either timeout or memory allocation issue
Level 4
Question for you, are you sure that the timeout is not from the email gateway you are using rather than the queue from Laravel? ie, with Amazon SES you get about 14 emails per seconds, anything above will cause problems and some of the queue jobs might get stuck.
Level 2
No emails are being generated, this is an on screen only notification just like you have here on Laracasts for logged in users. So you could click an alert bell icon after logging in and see a notification from the site owners for example saying upgrade your subscription before xxx date and the price goes up. There are groups of users with > 1,000,000 per group so just trying to identify the most effective way of firing the notification.
Level 6
@ritey Found a solution? I am dealing with the same thing,.
Level 1
@stgbuc I found the solution, use the Job Batch feature. i'am dealing with million row users.
Level 1
@mrrfly How do you send notifications to all users?
Level 1
I have 60,000 user data and chunking it into 1,000, then dispatching a Batch Job still times out (more than 60 seconds).
Level 25
@triadi In the queue config file, depending on which queue driver you are using, there is a retry_after setting. If this value is 60, this means that if the job is not finished in 60 seconds, it probably failed and it should try again.
If your queued job is taking longer than the value of this setting, you will receive errors. Try setting this to be a much higher value if your job is going to take a long time.
Level 1
well my users data is more than 1 million, and successfully send 65.500+ data an hour. Every device has different perform,
My device:
- Mac Pro M1
My Tech:
- Redis
and here a sample code, first, i create a job file to creating Job Batch
Bus::batch(new ProcessSendNotificationToUser($notificationCenter))
->name("Notification: {$notificationCenter->title}")
->onQueue('notifications')
->dispatch();
inside job file, i using lazy method to get all users data, and add other batch
foreach($users->lazy() as $user) {
$this->batch()->add(function () use ($user, $notificationCenter) {
$user->notify(new SendNotificationCenterToUser($notificationCenter));
});
}
however, i still improve this logic and code. maybe anyone have a great idea to improve my code
Level 50
I'd also look into the amount of data being passed into your job classes. For example, I had a job as follows:
php CompleteOrder::dispatch(order: $order);
$order was a full Eloquent model instance, which is large at the correspondiong table contained 58 rows.
As my complete order job literally only required the order id, I updated my job according:
php CompleteOrder::dispatch(orderId: $order->getkey());
It means you're passing much less data into your Redis queue.
Please or to participate in this conversation.