Laravel Scout with external data

Hello, I'm using Laravel Scout with the TNTSearch driver. I've been asked about the feasibility of integrating an external data source into Scout: I'll be provided with an array of data via an API from a CRM that I'll need to index in the Scout engine.

Now the question is: is this feasible? I've tried creating a dummy model and cycling through the array with a cron job, specifying $element->searchable(), but I always end up with an empty collection even though the .index file is generated correctly.

Am I missing something in the logic or is this simply not possible?

kevinbui

2 years ago

Level 41

It is possible.

Do you have your models in the DB and send an API request to obtain extra data for them? Or do you hydrate your models completely from external API calls?

For the former, this might work:

public function toSearchableArray()
{
    $response = Http::send(...)
	    ->json();

    $this->setAttribute('external_field_1', $response['field_1']);
    $this->setAttribute('external_field_2', $response['field_2']);

    return $this->toArray();
}

For the latter it gets a bit complicated, we cannot queue the action of syncing data, because jobs cannot reload models from the DB.

Do I understand you correctly? You are welcomed to elaborate.

FrancescoMansi

2 years ago

Level 1

@kevinbui Hi Kevin, thank you first of all for your response. My case is the second one, I have an API that provides me an array with title and description and I wanted to implement Scout on this array.

I have a Model defined as follows:

class ExternalTicket extends Model
{
    use Searchable; 

    protected $fillable = ['title', 'description', 'id']; 

    public function toSearchableArray()
    {
        return [
            'id' => $this->id,
            'title' => $this->title,
            'description' => $this->description
        ];
    }

    public function searchableAs()
    {
        return 'external_tickets';
    }
}

And before triggering the search, I launch a method to store the data in the .index file:

public function apiGetTickets()
    {
        $apiData =  $this->apiTraitGetTickets();

        foreach ($apiData as $data) {
            $externalData = new ExternalTicket();
            $externalData->fill($data);
            $externalData->searchable();
        }
        return true;
    }

Of course, the error on the screen is "SQLSTATE[42S02]: Base table or view not found: 1146" and hence the question whether it was possible to do something like this

kevinbui

2 years ago

Level 41

@FrancescoMansi I got that. I have three solutions in mind.

FIRST APPROACH: Use TNT Search APIs Directly

Forget about Laravel Scout, just use tnt search APIs directly to sync data. This is the most straightforward one and I recommend this.

SECOND APPROACH: Use Laravel Scout, But Do Not Queue Any Actions

In this scenario, I believe when we queue a sync action, a job cannot rehydrate the model object. Because that model object does not exist in the database.

I don't recommend this. Even if this works, inability to queue those actions is a huge letdown.

THIRD APPROACH: Use Scout With Sushi.

Laravel Sushi supports hyrating model objects from API responses. Pls have a look, maybe that works for you.

Please or to participate in this conversation.