illichov11's avatar

Set limit for queued jobs

I need to implement processing of big (different) amount of products - take them from one api and then send it to another api. Problem is that api where i should send this products has an limit (also different) - 2/4/etc requests per second.

I test how can i do it with this way:

call jobs like this:

collect(range(1, 100))->each(function ($i) {
            TestJob::dispatch($i);
        });

for my job i downloaded spatie/laravel-rate-limited-job-middleware package from composer

and here is my job class:

class TestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Batchable;

    protected $i;

    public function __construct($i)
    {
        $this->i = $i;
    }

    public function handle()
    {
        Log::alert($this->i);
    }

    public function middleware()
    {
        $rateLimitedMiddleware = (new RateLimited(false))
            ->allow(2)
            ->everySeconds(1)
            ->releaseAfterSeconds(2);

        return [$rateLimitedMiddleware];

    }
}

and here what i got in my log file:

[2024-04-17 12:43:12] local.ALERT: 1  
[2024-04-17 12:43:12] local.ALERT: 2  
[2024-04-17 12:43:16] local.ERROR: App\Jobs\TestJob has been attempted too many times or run too long. The job may have previously timed out. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): App\Jobs\TestJob has been attempted too many times or run too long. The job may have previously timed out. at /home/illichov11/projects/pims-dev.local/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:746)
[stacktrace]

(this error with trace reapeted for 98 times - for all anothers jobs)

But i need to something like pause processing another jobs, so that only 2 jobs will processed in 1 second, then - another 2 etc...

I am using database queue connection (my db is mariadb)

0 likes
5 replies
illichov11's avatar

@mabdullahsari I tried this before but it doesn't work properly as I need...

how this method works:

  1. i cant set limit per second - minimum that available - per minute
  2. i cant set dynamical amount of limit per something, only define it with constant value in AppServiceProvider
  3. i set it by this way just to test:
public function boot()
    {
        RateLimiter::for('jobs', function (object $job) {
            return Limit::perMinute(5);
        });
    };

class TestJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Batchable;


    protected $i;

    public function __construct($i)
    {
        $this->i = $i;
    }

    public function handle()
    {
        Log::alert($this->i);
    }

    public function middleware()
    {
        return [new RateLimited('jobs')];
    }
}

and here is my result in logs:

[2024-04-17 14:21:14] local.ALERT: 1  
[2024-04-17 14:21:14] local.ALERT: 2  
[2024-04-17 14:21:14] local.ALERT: 3  
[2024-04-17 14:21:14] local.ALERT: 4  
[2024-04-17 14:21:14] local.ALERT: 5  
[2024-04-17 14:22:17] local.ERROR: App\Jobs\TestJob has been attempted too many times or run too long. The job may have previously timed out. {"exception":"[object] (Illuminate\Queue\MaxAttemptsExceededException(code: 0): App\Jobs\TestJob has been attempted too many times or run too long. The job may have previously timed out. at /home/illichov11/projects/pims-dev.local/vendor/laravel/framework/src/Illuminate/Queue/Worker.php:746)
[stacktrace]

i got first 5 jobs processed in one moment and all another jobs fall as time out error

illichov11's avatar

@etiennedeschenes Hi, thanks for your help! this looks like a similar solution to defining public $tries in the Job class. But in my case, there might be too many thousands of tasks in the queue (with a large run-up depending on the task), so it will take a very long time and i am not sure that it will be good practice to set like few hours for retryUuntil or few thousands for tries

emergingdzns's avatar

@illichov11 Did you ever find a good solution? I'm having the same issue. We use SQS for our queue and all of the built-in Laravel functionality only allows per minute, but the API we are calling has a per second restriction that I need to be able to handle. Every time I try to engage the RateLimited class, even a custom class, I end up with the same error about max retries and such.

Please or to participate in this conversation.