How do I improve bulk mailing performance with Redis Queues?
My application needs to send a digest to about ~50000 subscribed members and I've been able to setup the Redis Queues and Horizon to monitor the job.
Because each user receives their 'custom' bulletin, I can only process one mail at a time. The overall mail sending takes up lot of time - ~ about 30-40 minutes.
I'm wondering if there's a way to improve the dispatch times. I'm of course using SES to process mails and my limit is 500 emails/second.
What are the ways I can improve my email delivery?
The best way to improve email delivery is not to use your sever to send it. You have absolutely no control over 50.000 users out of which some do not even know how email works. 200 of them will mark you as a spammer and the troubles will start. Use some outside service to queue and send emails. Otherwise, you will be soon looking at getting new IP for your server.
There are many services out there that offer this kind of products.
My subscriber base comprises of engineers who have opted-in to receive the digest; and we have pretty good email open rates. I think reception of email is not an issue at all. The bounce rate and unsubscribe rates are super low too.
My question is from the performance point of view. So far, I've set up Redis powered queues that deliver the mail to SQS for final delivery; and I think it works pretty well. My SQS send rate is pretty good too (for my requirements) at 500 emails/second.
I however do not know how could I make best use of this setup to speed-up my email delivery. Right now, I'm chunking users in batches of 300 and then running a foreach loop to send email to SQS.
I'm wondering if there's anything I could do better.
@thebigk Well Laravel can only send one email at the time right. So I think you have two options here. Either run more workers so you can process more jobs from the queue at the same time. I would probably run those workers on separated servers.
Option two is using a third party mail delivery services where you can just send an array of email addresses and they will then figure out how to send the emails as fast as possible with their own internal queue. This way you don't have to think about it and it also doesn't impact the performance on your own webserver ;)