Is there a way to add jobs to a batch one by one or in chunks?

@Glukinho thank you! Here's what I've tried:

$files = File::take(10)->get();
$batch = Bus::batch([]);
foreach ($files->chunk(5) as $chunk) {
	$jobs = $chunk->map(fn ($file) => new MyJob($file))->all();

    $batch->add($jobs)->dispatch();
}

Expected behavior: it adds 10 jobs total (5 + 5 each SQL request);

Observed behavior: it adds 15 jobs total (5 + 10);

@Glukinho In your example, I see three more jobs ->add()ed after dispatching. But ->add() itself does not persist anything to the database, as far as I understand… when they are persisted exactly?

@pilat again:

$pending_batch = Bus::batch([]); // returns Illuminate\Bus\PendingBatch object
$pending_batch->add(new TestJob); // job is appended to pending batch and will be actually queued after ->dispatch() execution

$dispatched_batch = $pending_batch->dispatch(); // returns Illuminate\Bus\Batch object
$dispatched_batch->add(new TestJob); // job is actually queued to database

You need to add jobs to a dispatched batch, AFTER you executed ->dispatch().

$files = File::take(10)->get();

$batch = Bus::batch([])->dispatch();  // dispatch empty batch

foreach ($files->chunk(5) as $chunk) {
	$jobs = $chunk->map(fn ($file) => new MyJob($file))->all();

    $batch->add($jobs); // no need to ->dispatch() here, the batch is already dispatched, jobs go right to queue as soon as they are added to the batch
}

@Glukinho Ah, I see. I need to use separate $dispatched_jobs variable to add more jobs into the batch. This way I can see all the jobs actually added to the database in proper amount, but the script itself crashes with "Allowed memory size of 134217728 bytes exhausted" error.

Looks like it still tries to load all the jobs into memory at some moment of time :(

@Glukinho

Here's what I've tried in Tinkerwell:

$s = FilesSession::first();

$batch = Bus::batch([]);
$files = $s->files()->notUploaded();
$isFirst = true;
$dispatchedBatch = null;
foreach ($files->lazyById(100) as $file) { // I've tried ->cursor() as well -- still not enough memory :(
    if ($isFirst) {
        $batch->add(new UploadOneFileToStorageJob($file));
        $dispatchedBatch = $batch->dispatch();
    } else {
        $dispatchedBatch->add(new UploadOneFileToStorageJob($file));
    }

    $isFirst = false;
}

This code runs for some time, there are jobs in the database (as expected) in the results. But in the end this code fatals because of the memory limit. Not sure where is the leak exactly…

@pilat show the actual code, and job class too. Do you try to put file content into a job, or just file path?

I have added 100 000 jobs to queue now from tinker the way I showed, 100 000 jobs showed up in a database jobs table, memory consumption was about 16 Mb all the time:

> $batch = Bus::batch([])->dispatch()
> for ($i = 0; $i < 100_000; $i++) { $batch->add(new App\Jobs\TestJob(0)); echo "$i\n"; }
1
2
3
...

@pilat DON'T call ->dispatch() every time in a loop. Call ->dispatch() one time on an empty batch before the loop:

$batch = Bus::batch([])->dispatch();

Then, in the loop, just add a job to a batch. The batch is already dispatched, so each job goes to the queue immediately and is not collected in a batch object (so memory is not consumed):

foreach (...) {
	$batch->add(new UploadOneFileToStorageJob($file));

	// and DON'T call any ->dispatch() in a loop
}

@pilat

But ->add() itself does not persist anything to the database, as far as I understand

If a batch is pending (->dispatched() wasn't executed), yes. On a dispatched batch (on which ->dispatched() was executed) - no, a job is persisted to a queue (to a database in your case).

@Glukinho Every $file is an instance of an Eloquent model. No actual file content in it. And I don't run any queue workers at this stage, just trying to find a way to schedule a big batch :)

@pilat Ok, do as I wrote before, it will work.

Update: queue worker is not related, it can be running or stopped, no matter.

@Glukinho considering that it actually adds all of the jobs to the DB, I assume it was something in Tinkerwell that run out of memory. In particular, I can see that it tries to collect all the queries, even though I've specifically chosen to use CLI Output, not the Detail Dive pane…

...

Yup, if I run the same code in plain old php artisan tinker, it would not fail.

Thank you!!

SUMMARY:

1. When dispatching a batch for the first time (with a single or manageable amount of jobs), assign the result to $dispatchedBatch variable: $dispatchedBatch = Bus::batch($firstChunkOfJobs)->dispatch();
2. Add more jobs to the DISPATCHED, instead of PENDING batch: $dispatchedBatch->add($extraJobOrJobs);

@pilat yes, you got it right.