Be part of JetBrains PHPverse 2026 on June 9 – a free online event bringing PHP devs worldwide together.

gitwithravish's avatar

Laravel Scheduler Scaling

Each user can setup his own schedule commands in our application. The schedule commands must be executed immediately at given point. When you consider the varying schedules, the start of any given hour and at midnight on any given day, scaling to support this influx of traffic is being a challenge.

For example, currently following is the cron job execution based on time.

00:00 => 529 Crons

01:00 => 120 Crons.

02:00 => 0 Crons.

03:00 => 37 Crons.

04:00 => 225 Crons.

and so on...

These numbers change with time. The cron should execute on exact given time matters alot.

We can use queues but when there is a heavy number of crons scheduled for any given time, task are delayed in execution because queue executes tasks one by one. We can increase the number of queue workers but this is also not going to solve the problem.

The question is how do we scale our architecture (Dockerised kubernetes cluster). Currently the machine runs out of cpu and ram by 00:00 hour. and sits ideal unnecessarily when there are no tasks to run.

0 likes
2 replies
LaryAI's avatar
Level 58

One solution could be to use a distributed task scheduler like Celery or Apache Airflow. These tools allow for distributed task execution across multiple machines, which can help with scaling and handling large numbers of scheduled tasks. Additionally, they provide features like task retries and error handling, which can help ensure that tasks are executed successfully.

Another option could be to use a cloud-based solution like AWS Lambda or Google Cloud Functions. These services allow for serverless execution of code, which can help with scaling and handling large numbers of scheduled tasks. They also provide features like automatic scaling and pay-per-use pricing, which can help reduce costs and improve performance.

In terms of optimizing the current setup, one approach could be to use a load balancer to distribute traffic across multiple machines. This can help with scaling and handling large numbers of requests. Additionally, optimizing the code and database queries can help reduce resource usage and improve performance.

Here's an example of how to use Celery with Laravel:

# Install Celery
pip install celery

# Create a Celery task
from celery import Celery

app = Celery('tasks', broker='pyamqp://guest@localhost//')

@app.task
def add(x, y):
    return x + y

# Call the task from Laravel
Route::get('/test', function () {
    $result = app('Illuminate\Contracts\Bus\Dispatcher')->dispatch(new \App\Jobs\CeleryTaskJob());
    return $result;
});

# Define the job in Laravel
namespace App\Jobs;

use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class CeleryTaskJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        $result = \Http::get('http://localhost:5000/add/1/2')->json();
        logger($result);
    }
}
Snapey's avatar

I guess you are failing to attain execution of all jobs at the right time currently, so this begs the question, how important is the precise scheduling of the job?

I would first look for options to contractually stagger the jobs. You could for instance only allow so many jobs per minute when allowing the crons to be setup and tell the user that there is no capacity at certain minutes.

You could advise users that their job will be started randomly within a specified window.

Then, from a performance point of view, you have a certain amount of work to do. You can either;

  • scale-up resources to cope with the load
  • reduce the work so that the job executes quicker

Larry suggested scaling out the capacity. Having simple lambda functions that can be launched in parallel could be a solution but is technically tricky when you consider the work to get the job to the lambda and getting the results back.

Reducing the work might be an option if there is anything you can do in advance of the runtime to prepare the job, or if you can delay the processing/storage of the result (difficult without specifics).

You can also look at the most efficient queue processing methods. For instance, using database queues can be relatively slow because of the overhead of managing the queue itself (extra DB read and writes)

Just some thoughts. I can't imagine there is a 'correct' answer.

Please or to participate in this conversation.