Your approach is solid and is a common pattern to reduce queue latency in high-throughput scenarios. By batching the job dispatch into a single SQS call, you’re minimizing the network round-trips and the overhead of multiple SQS requests per web request. This is especially helpful when you’re on Laravel Vapor, where you don’t have access to after-response hooks.
A few thoughts and suggestions:
1. Your Approach is Sound
- Batching jobs into a single "bucket" job is a good way to reduce SQS latency.
- Using
SerializableClosureis a smart way to capture the work to be done, especially when the jobs are not always serializable themselves.
2. Potential Improvements
a. Use Real Jobs, Not Closures (If Possible)
While closures are flexible, they can make debugging, retrying, and monitoring harder. If you can, consider collecting actual job instances (that implement ShouldQueue) instead of closures. This way, you retain all the benefits of Laravel's job system (like retries, tags, etc.).
Example:
// Instead of:
QueueBucket::add(fn () => SendNewUserAdminNotification::send($user));
// Do:
QueueBucket::add(new SendNewUserAdminNotification($user));
Then, your QueueBucketJob would simply dispatch each job:
public function handle()
{
$this->jobs->each(fn ($job) => dispatch($job));
}
b. Serialize Models Carefully
You mentioned double hydration. If you’re passing Eloquent models to jobs, they will be serialized and re-hydrated. To minimize this, always pass only the model IDs to jobs, and re-query in the job’s handle() method.
c. Consider Chunking for Large Buckets
If you ever have hundreds of jobs in a single request, you might want to chunk them into multiple bucket jobs to avoid hitting SQS or Lambda payload limits.
$chunks = $this->jobs->chunk(50);
foreach ($chunks as $chunk) {
dispatch(new QueueBucketJob($chunk));
}
d. Middleware Placement
You’re already using middleware to call QueueBucket->run(). Make sure it’s at the end of the middleware stack so it runs after all work is done.
3. Example Refactor
Here’s a version using job instances:
QueueBucket.php
namespace App\Domain\Shared\Jobs;
use Illuminate\Support\Collection;
class QueueBucket
{
private Collection $jobs;
public function __construct()
{
$this->jobs = new Collection();
}
public function add($job): void
{
$this->jobs->push($job);
}
public function run(): bool
{
if ($this->jobs->isNotEmpty()) {
dispatch(new QueueBucketJob($this->jobs));
$this->jobs = new Collection();
return true;
}
return false;
}
}
QueueBucketJob.php
namespace App\Domain\Shared\Jobs;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Collection;
class QueueBucketJob implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, SerializesModels;
public function __construct(
private Collection $jobs
) {}
public function handle(): void
{
$this->jobs->each(fn ($job) => dispatch($job));
}
}
Usage:
QueueBucket::add(new SendNewUserAdminNotification($user));
QueueBucket::add(new UpdateEmailListSubscriberAction($user));
QueueBucket::add(new AccountEmailVerificationNotification($emailRecord->new_email, $emailRecord->verification_code));
4. Tradeoffs
- Pros: Drastically reduces SQS latency per request.
- Cons: Slightly more complex, and jobs are not immediately visible in the queue (they’re nested in a bucket job).
5. Conclusion
Your solution is good and is a proven pattern. If you can, prefer passing job instances over closures for better integration with Laravel’s queue system. Otherwise, your current approach is perfectly valid for your use case.
Let me know if you want a more detailed implementation or have any follow-up questions!
