Delay between processes in Job Queue

Delay the job: https://laravel.com/docs/5.2/queues#delayed-jobs or just use the pause() function of PHP.

Check out the "delayed jobs" in the documentation: https://laravel.com/docs/5.1/queues#delayed-jobs

Assuming you only have one queue listener, you could delay each job by 60 seconds so you only execute 60 API calls an hour.

Hi there,

thanks for your suggestions. Here is what I've tried:

<?php

namespace App\Http\Controllers;

use App\Condition;
use App\City;
use DB;
use App\Jobs\FetchConditions;

class ConditionsController extends Controller
{
    public function fetchConditions() {
        $stations = Station::all();

        foreach($stations as $station) {
            $job = (new FetchConditions($city))->delay(1);
            $this->dispatch($job);
        }

        return 'Fetching Conditions...';
    }
}

When I run the queue, it still execute 3/4 jobs each seconds. Something wrong on my code?

The delay argument takes the number of the seconds you wish the job to be delayed by. You also may have more than one queue listener running on your server. You may want to think about making it only one so you don't hit your API limit.

@TylerODonnell Yes I know, I've choosen one second to excecute 60 jobs in a minute (with one worker). However that isn't happening (still get 3/4 jobs excecuted each second). Any idea?

If your delay is in a foreach loop yes you will get many jobs per second still. Think about it for a second. Your loop runs 10 times a second each delay is one second, the 10 iterations will fire after one second causing all 10 to fire microseconds from each other.

You need to add a counter (suggest a for instead of a foreach loop) to add one second to your delay for each iteration of the loop.

Or add sleep() to the bottom of your loop to delay the next iteration one second and remove the delay(1).

@jekinney, but that will delay the insertion of the job in the queue, I need to be able to define an execution limit on the queue side. Maybe Redis has some configuration to achieve that? I didn't found anything about it. My way to approach the problem could be most likely be the not the correct one. Maybe there are other ways than using a Queue to limit API request to 60 per minute.

Finally I think I misunderstood how Queues works. I think there is no control on how much jobs should the queue execute each seconds. I was using the ->delay(1) wondering why didn't worked, but the reality is that this function delays only the execution of the first job in the queue. To achieve what I needed, I had to modify my previously submitted code this way:

<?php

namespace App\Http\Controllers;

use App\Condition;
use App\City;
use DB;
use App\Jobs\FetchConditions;

class ConditionsController extends Controller
{
    public function fetchConditions() {
        $stations = Station::all();
    $this->dispatch(new FetchConditions($stations));

        return 'Fetching Conditions...';
    }
}

Using the foreach in the Job class using the suggested sleep(1) method:

<?php

namespace App\Jobs;

use App\City;

class RetrieveStations extends Job
{
    protected $cities;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct ($cities)
    {
        $this->cities = $cities;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {

        foreach($this->cities as $city) {
            
        // code here

            sleep(1);
        }

    }
}

Now the job runs in background and the request to the Weather API will be execute each second.

Thank you all for the help!

@ampersand Not a good practise to make your process sleep. Try this ,

class ConditionsController extends Controller
{
    public function fetchConditions()
     {
        $stations = Station::all();
        
        $i = 0;
        $numberOfJobsPerSecond = 4;
    
        foreach($stations as $station)  {
            $delay = (int) floor($i++ / $numberOfJobsPerSecond);
            $job = (new FetchConditions($city))->delay($delay);
            $this->dispatch($job);
        }
    }
}

@d3xt3r the suggested code still execute 2-4 jobs per second, even if I define $numberOfJobsPerSecond = 1;.

I had the very same problem but even sleep wasn't enough for me because if 10 users did say update at the same time then sleep will sleep them at the same time and the delay start at the same time therefore the execution happens at the same time. i.e too much request to the API.

What i did is this.

    public function update(Request $request, $id)
    {
            $delay = \DB::table('jobs')->count()*10;

            $queue = new Update($request->all(), $id, \Auth::user()->email);
            $this->dispatch($queue->delay($delay));
     }

in this case if the queue is empty then delay is 0. If the queue has Y items then next item will be delayed by (Y*10). 10 sec from the last item.

@Ali_Ali I'm facing same issue with Amazon API and I also agree that this is the best way to go :)

Maybe this? ^_^

$orders = \App\Models\Order::take(2)->get();
            $i = 5;
            foreach ($orders as $order){
              
                dispatch((new \App\Jobs\SendTracknumberEmail($order))->delay($i));
                $i = $i + 10;
              
            }

I know how to set a delay for a job, but how can I configure the delay for a queued event listener? I'm using laravel 5.5.

I've used the release() function in the Job Handler function.

In my case I needed the job to execute every 10minutes, sleep and delay didn't work as it considered that Job was running when it was just sleeping. It's good if you want to retry after certain amount when API fails.

   public function handle()
        {
        if($badAPI){
            $this->release(600); // releasing back to queue in 10 mins
        }
    }

In situation like this, you could switch to using cron and schedule a check every minute. Inside that check, you could do the for loop with sleep

Try this

<?php

namespace App\Http\Controllers;

use App\Condition;
use App\City;
use DB;
use App\Jobs\FetchConditions;

class ConditionsController extends Controller
{
    public function fetchConditions() {
        $stations = Station::all();

	$now = now();
        foreach($stations as $station) {
            $now = $now->addSeconds(1);
            $job = (new FetchConditions($city))->delay(now );
            $this->dispatch($job);
        }

        return 'Fetching Conditions...';
    }
}

Try the rate limiting middleware:

https://laravel.com/docs/9.x/queues#job-middleware and https://laravel.com/docs/9.x/queues#rate-limiting