[Laravel Scout - Meilisearch] Mass Importing Dataset Using Queue

Hello there folks!

I'm trying to eager load the models relationships before importing the dataset into the engine, and when I do so without queueing the job, it works like a charm, but once I set the SCOUT_QUEUE variable to true, the job fails.

Here is the code snippet:

   protected function makeAllSearchableUsing($query)
   {
      return $query->with('brands.brand');
   }
   
   public function toSearchableArray()
   {
      $brands = $this->brands->pluck('brand.name')->implode(' - ');

      return [
         'id'           => (int) $this->id,
         'name'         => $this->name,
         'name_short'   => $this->name_short,
         'slug'         => $this->slug,
         'brands'       => $brands,
      ];
   }

I'm aware that queue workers operate in a separate processes, therefore Lazy loading is not an option, so I believe the option I'm left with is to load the relationships from within the toSearchableArray method, like this:

public function toSearchableArray()
{
   $this->load('brands.brand');

   $brands = $this->brands->pluck('brand.name')->implode(' - ');

   return [
      'id'           => (int) $this->id,
      'name'         => $this->name,
      'name_short'   => $this->name_short,
      'slug'         => $this->slug,
      'brands'       => $brands,
   ];
}

For one or a small number of models, it can be fine, but using this approach to import hundreds of models can result into multiple database queries, and that's inefficient.

If anyone has encountered this issue before and found a good solution/workaround for it, please share it with us as it will help alot!

Your assistance is much appreciated, and thanks in advance!

LaryAI

3 years ago

Level 58

One solution to this issue is to use the chunkById method to retrieve the models in batches and then eager load the relationships before indexing them. Here's an example implementation:

protected function makeAllSearchableUsing($query)
{
    return $query->with('brands.brand');
}

public function queueMakeAllSearchable()
{
    $self = $this;

    $this->newQuery()->chunkById(100, function ($models) use ($self) {
        $models->load('brands.brand');
        $self->makeAllSearchable($models);
    });
}

In this example, we're using the chunkById method to retrieve the models in batches of 100. We then load the relationships for each batch of models before passing them to the makeAllSearchable method to be indexed.

To queue this job, you can add a new method to your model that calls the makeAllSearchable method in a queued job:

public function queueMakeAllSearchable()
{
    $this->with('brands.brand')->searchable();
}

Then, in your code, you can call this method to queue the job:

$model->queueMakeAllSearchable();

This will enqueue a job to index all of the models in the background using the SCOUT_QUEUE setting.

Please or to participate in this conversation.