Sending Batch/Bulk Emails

I have a question about sending batch/bulk emails using by dispatching a Job to a queue.

The gist of it.

When a destination get's created and saved I dispatch a job.

public function saved(Destination $destination): void  
{  
    if ($destination->wasRecentlyCreated) {  
        dispatch((new SendEmailToUsersJob($destination->id))->onQueue('sendEmailsToUsers')->delay(now()->addSeconds(5)));  
    }}

In the Job I'm gathering all the data I need to send the email to the Users and using CHUNK to email 10 users. I use a Foreach Loop and add SLEEP 60 so I'm only sending 10 emails every 60 seconds.

public function handle(): void
{
    $destination   = Destination::where('private', 0)->findOrFail($this->destinationId);
    $emailCampaign = EmailCampaign::where('is_active', 1)->where('slug', 'email-to-users')->first();
    
    if ($destination && $emailCampaign) {
    
    Sponsorship::whereIn('value_id', function ($query) use ($programIds) {
    $query->select('id')
    ->from('values')
    ->whereIn('program_id', $programIds);
    })->with(['value', 'user'])
    ->orderBy('id')
    ->chunk(10, function ($sponsorships) use ($emailCampaign, $destination) {
    // Code to get data to pass to email for each batch of 10 sponsorships  
    $sponsorshipsByUserAndTown = $sponsorships->groupBy(function ($sponsorship) {
    return $sponsorship->user_id . '-' . $sponsorship->value->town_id;
    });
    foreach ($sponsorshipsByUserAndTown as $sponsorshipsForUserAndTown) {
    // data to pass to the email...  
    
    Mail::to($sponsor->email)->send(new EmailToUsersEmail(
    $sponsor,
    //.. email data  
    ));
    }
    // Pause for 1 minute after each batch of 10 emails  
    sleep(60);
    });
    }
}

The issue I'm running into is It send the first 10 fine, but then timeout when 60 seconds passes and the next batch of 10 emails need to send. And the Job fails. When I rerun the job, to no surprise it emails the same 10 Users again and then fails.

What is the proper way to do this?

Would it be better to Place each email in it's own Job? Based on the users that would be around 2000+ jobs.

I've also read a little about Dispatching a Batch and it sounds like I would create all those 2000 jobs then place in a single batch.

Dispatching a Batch: To ensure that we only execute a single store command that contains all our jobs, we may dispatch a batch:

use Illuminate\Support\Facades\Bus;
use App\Jobs\RefundAttendee;
$jobs = $this->conference->attendees
->map(function($attendee) {
return new RefundAttendee($attendee)
});
Bus::batch($jobs)->dispatch();

Using batch() instructs the bus to collect and store all the jobs in a single insert query."

This requires creating a job_batches table: php artisan queue:batches-table

That seems a little to complex for my needs though.

I'm not looking for someone to write this code for me, rather I need a solid solution from someone with experience in this area. Looking for the "Best Practices" for something like bulk/batch email sending.

FYI: I'm on Laravel 10 and using the Database (No Redis)

LaryAI

3 years ago

Level 58

One solution to this problem is to use Laravel's built-in queue system to handle the email sending in the background. Instead of sending all the emails at once, we can dispatch a job for each email and let the queue handle the rest. This will ensure that the emails are sent in a timely manner without causing timeouts or other issues.

Here's an example of how this can be done:

Create a new job for sending emails:

php artisan make:job SendEmailJob

In the handle method of the job, send the email using Laravel's Mail facade:

public function handle()
{
    Mail::to($this->user->email)->send(new EmailToUsersEmail($this->user, $this->data));
}

In the saved method of the Destination model, dispatch a job for each user:

public function saved(Destination $destination): void  
{  
    if ($destination->wasRecentlyCreated) {  
        $users = User::all(); // or any other way to get the users you want to email
        foreach ($users as $user) {
            dispatch(new SendEmailJob($user, $destination->id))->onQueue('sendEmailsToUsers');
        }
    }
}

Configure your queue driver to use the database driver:

QUEUE_CONNECTION=database

Run the queue worker to process the jobs:

php artisan queue:work

This will process the jobs in the sendEmailsToUsers queue and send the emails in the background. You can also configure the queue worker to use multiple processes or threads to speed up the processing.

Using this approach, you don't need to worry about timeouts or other issues related to sending bulk emails. The queue system will handle the emails in the background and ensure that they are sent in a timely manner.

daugaard47

3 years ago

Level 7

I appreciate the reply Quickdraw, but would love a humans take on this.

Please or to participate in this conversation.