O.Zadorozhnyi's avatar

Slow seer on inserting 100k records by using Eloquent

Hey, I have two eloquent models User and Image with has Many relation in database . I need generate 100k Images and assign it to users RANDOMLY.

Now my seeder class looks like:


declare(strict_types=1);

namespace Database\Seeders;

use Generator;
use Illuminate\Database\Console\Seeds\WithoutModelEvents;
use Illuminate\Database\Seeder;
use App\Models\User;
use App\Models\Image;
use App\Traits\Chunkable;

class ImageSeeder extends Seeder
{
  use Chunkable;

  /**
   * Run the database seeds.
   */
  public function run(): void
  {

    foreach ($this->chunk(generator: $this->images(20000), size: 1000) as $chunk) {
      echo 'lol' . PHP_EOL;
      Image::insert($chunk);
    }
  }
  private function images(int $count): Generator
  {
    $ids = User::pluck('id')->toArray();
    $now = now();

    for ($i = 0; $i < $count; $i++) {
      yield array_merge(
        Image::factory()->make()->toArray(),
        [
          'user_id' => $ids[array_rand($ids)],
          'created_at' => $now,
          'updated_at' => $now,
        ]
      );
    }
  }
}

Chunkable.php trait code:


declare(strict_types=1);

namespace App\Traits;

use Generator;

trait Chunkable
{
  /**
   * Chunks a generator.
   *
   * @param Generator $generator
   * @param integer $size
   * @return Generator
   */
  private function chunk(Generator $generator, int $size = 100): Generator
  {
    $chunk = [];
    foreach ($generator as $item) {
      $chunk[] = $item;
      if (count($chunk) === $size) {
        yield $chunk;
        $chunk = [];
      }
    }
    if (!empty($chunk)) {
      yield $chunk;
    }
  }
}

This code is very slow (around 1 sec on one SQL INSERT for 1000 recodrs in chunk) I need some ideas how to improve this code. Thanks

0 likes
2 replies
Ssionn's avatar

@o.zadorozhnyi In this case you'd wanna use batches/jobs. Also preferably use a bunch of arrays and merge them as someone on stackoverflow said:

Generators weren't designed to replace arrays. They were intended as a way to reduce boilerplate code when implementing Iterator objects. Generators will always be slower than arrays, because generator has to generate values each time you call next()

O.Zadorozhnyi's avatar

@Ssionn tnx, thinking about jobs, but hope I can find solution via Eloquent lazy loading + generators, some like that

Please or to participate in this conversation.