Robust Mass Email with Mandrill

It's perfectly fine to do it like this. Like you said, each email is unique so you can't bulk send them to multiple users at once!

Just be careful when sending this amount of emails through Mandrill that you don't get yourself a bad reputation and throttled due to non deliveries, rejected mail and spam reports.

That probably depends on how recently you acquired the emails in your system and how many emails you'd send from the start. I'd probably recommend building the number of emails up over time, rather than starting to send so many emails suddenly. And try to spread the emails out over as much time as possible.

How have you set the beanstalk queue up? Would you queue the potentially thousands of emails all at once? And just let them send as quick as they can? That would possibly lead to the bad rep and being throttled.

It might be worth setting up a cron job to call a custom artisan command every x minutes. This command would then push a certain number of emails to the queue. Should reduce the likelihood of it causing problems but doesn't mitigate the risk entirely. I'd keep a careful eye on Mandrill and the stats it provides.

EDIT: You'll need to store the last position (userid) somewhere so that your artisan command knows which emails to queue next. That could either be in the database or in the cache, as an example.

Thanks @bobbybouwmann and @Rjs37 , I will set up that custom artisan command, that sounds like a great idea!

Any ideas about the number of emails I should do for each batch and the interval of the cron job? Thanks again!

Will definitely keep an eye on my Mandrill rep. I might reach out to them regarding this, if I do and hear back I'll post here.

No worries @StormShadow

I've added a quick edit to my previous post concerning how to know where to continue from with your artisan command.

Those insights came mainly as I did something similar myself several months ago for a one off email to our users. Unfortunately we still got throttled after a few days, the emails (from our own users) were collected over a number of years and so the delivery / bounce rates were pretty low / high respectively. A couple of months later and the reputation is still classed as Poor. So be very careful. I think the recovery rate depends on how many good emails you send out to balance out the 'bad' ones.

I think that delivery %, bounce % and the spam rate (users/servers reporting your emails as spam) are the primary causes of being throttled. So be sure the emails were acquired and validated fairly recently, and that you've been given their permission to send them these monthly newsletters. As long as you do that and follow the advice above you should be okay. If you send a lot of normal system emails (activations, pass resets etc) then that will help balance out the %'s somewhat especially with smaller batches.

In terms of intervals I chose to do batches of 50 emails every 5 minutes. It depends on how many emails you need to send and what your current hourly limit is. Our hourly limit was around 1k at the time. Mandrill will increase your hourly limit (if you have an excellent rep), in response to recognising that you're sending more emails, but I wouldn't recommending going from sending 5 per day to 600 per hour, as an example.

Also make sure you account for failed jobs, we had some interesting situations of email addresses that had been validated but were being rejected by Mandrill.

@Rjs37 love it, thanks again. I can't believe I was just considering just dumping all my emails on the queue like a cowboy! Your post motivated me to do some more research into Mandrill. Currently we have an excellent reputation and can send 500 per hour, but I'm going to set up that artisan command as you suggested.

I'm curious did you set up something like an emails db table and then at a certain time via cron (once a month in my case) that added added the emails to the table and then had a separate artisan command that would send the 50 every 5 minutes and then delete them from the emails table? Are you using the mandrill API or SMTP ?

Yes we do send lots of transaction type emails, but now I'm thinking I should quickly develop this custom email queue and send all my emails via it so I can have greater control over my email rate, and stay on top of my mandrill rep.

Cheers!

@StormShadow As it was just a one off job I just had a temporary table with a single row storing the user id of the last email that was queued. I then just grabbed the next x users with a userid greater.

I think it could be overkill to micromanage your own queue in the database. Unless your conditions are very complex I'd grab the emails straight from the user record. I'd still recommend using the laravel queue functionality to send your transaction emails so that it doesn't slow down user requests.

This is what I'd be tempted to do: Single db table: emails - id, subject/title, last_user_id, completed

Your command could then search for any non completed emails, grab the last_user_id and carry on from where it left off. As an example, here's the fire function from my command:

public function fire()
    {
        $limit = $this->option('limit');

        // Find starting position from temporary table
        $skip_id = DB::table('emailtemp')->pluck('user_id');

        $this->comment('Fetching '.$limit.' users with an id greater than '.$skip_id.'...');

        // Count users
        $count = User::where('subscribed', '1')->where('activated', '1')->count();

        // Get next batch of users to process
        $users = User::where('subscribed', '1')->where('activated', '1')
            ->where('id', '>', $skip_id)->orderBy('id', 'asc')->take($limit)->get();

        $this->comment('Found '. count($users) .' users of '. $count .' total users');

        // Loop users and queue email
        if (count($users) > 0 ) {
            foreach ($users as $user) {
                $this->addtoQueue($user);
            }

            // Set finishing user id onto temporary table
            DB::table('emailtemp')->update(array('user_id' => $user->id));

            $this->info('Success! All user emails have been queued. Next loop will begin after id '.$user->id);
        }
        else
            $this->info('There are no more users left to email. Stop task');
    }

And then this was the command that I could run within the cronjob every 5 mins

php path/for/site/artisan queueemails --env=production --limit=50

EDIT: Oh and I was using SMTP at the time but I've just switched to the API as I'm in the middle of the upgrade to Laravel 5.

@Rjs37 Above and beyond the call of duty my friend! Good on ya mate! :)

@StormShadow No worries, If I can help someone else avoid getting throttled/blacklisted then it's my pleasure :). Good luck with it!