Dynamically rate limiting jobs on queue with the same name
We have a job that synchronizes user information with a 3rd party CMS used by the marketing team. We use events to listen for User model changes that require a re-sync and queue the job.
pseudo-code for the sake of example:
class CMSUserSyncListener {
public function userDidChangeName(User $user) {
dispatch(new CMSSyncJob($user));
}
}
class CMSSyncJob {
public function __construct(private User $user);
public function handle() {
// Use CMS SDK (Guzzle wrapper) to UPSERT $user data
}
}
The problem we face is rolling rate limits, with max requests per second, minute, hour & day. We don't fully know what these limits are, they change regularly. 3rd party support team just tell us that they don't publish the limits, they are based on per client "algorithms" and that our integration should monitor the Rate Limit headers from their API, and adjust accordingly.
So what we need is when a Job runs and get's a rate limited response, it notifies the queue to delay all future jobs of the same name/class.
We could use release($delay) but this results in the queue spinning through hundreds/thousands of jobs just to release them (I also read somewhere that this increments the fail counter, so if a job needs releasing several times, it could just fail).
The ideal would be a way to say to the queue "Any job with class CMSSyncJob in the queue, skip for the next few minutes".
Our only other idea is to have a separate node.js application that's just always active, monitoring a redis list for updates that need syncing (so CMSSyncJob would just need to drop the payload into a redis list instead of sending it). And then sleep in the node run loop whenever we hit a rate limit. But that massively over complicates our system as we don't have anything like that currently (we're pretty basic).
I know this is a long one, but thoughts?
Please or to participate in this conversation.