Problems implementing pdfphilip/laravel-elasticsearch due to pagination

I am facing a problem, where I cannot use the liberia pdfphilip/laravel-elasticsearch which basically extends the Eloquent model for using Elasticsearch indexes.

When I went to implement it I ran into the aggregations (I think) that arise in how Filament builds the model listing.

The point is that the package works with a method that allows you not to change the Elastic configuration (not recommended, especially when we are talking about hundreds of millions of indexed docs) and instead Chunking proposes the use of an internal field to use the chunking sorts on it, Model::chunkById

I have tried without success to change the primary key of the model but it doesn't help.

I've been looking at the possibility of doing something with this, but I haven't seen how.

Any ideas?

Error

Certainly, being able to work with the best library to use Elasticsearch, following the Laravel philosophy, is a great contribution.

LaryAI

2 years ago

Level 58

It seems like you're trying to integrate Elasticsearch with Laravel using the pdfphilip/laravel-elasticsearch package and are encountering issues with pagination, particularly when dealing with large datasets and Elasticsearch's limitations on deep pagination.

The error message you're seeing suggests that there's an issue with accessing field data on the _id field, which is disallowed by default in Elasticsearch for performance reasons.

To work around Elasticsearch's deep pagination limitations, the chunkById method is recommended. This method will paginate through records by their primary key in chunks, rather than trying to load all records at once, which can be problematic with large datasets.

Here's a general approach to implementing chunking with Elasticsearch in Laravel:

use App\Models\YourElasticsearchModel;

// Define the chunk size
$chunkSize = 1000; // Adjust the chunk size as needed

// Use the chunkById method to process large datasets
YourElasticsearchModel::chunkById($chunkSize, function ($models) {
    foreach ($models as $model) {
        // Process each model here
        // For example, you could index the model data into Elasticsearch
    }
});

If you've tried changing the primary key of the model and it hasn't helped, make sure that the primary key you're using is suitable for chunking. It should be a numeric, incrementing key that allows Laravel to efficiently retrieve the next chunk of records.

Regarding the error message you're encountering, if you absolutely need to enable field data access on the _id field, you can update the dynamic cluster setting indices.id_field_data.enabled to true. However, this is generally not recommended as it can lead to performance issues.

To enable field data on the _id field, you would need to execute the following command against your Elasticsearch cluster:

PUT /_cluster/settings
{
  "persistent": {
    "indices.id_field_data.enabled": true
  }
}

Please note that this change can have significant performance implications, and you should carefully consider whether it's necessary for your use case.

If you continue to have issues, it might be helpful to provide more specific code examples or error messages so that the community can offer more targeted advice.

abkrim

2 years ago

Level 13

Hello @LaryAI

The question was already answered by other AIs in that sense.

First of all, it does not seem to be a problem just with Filament but with Livewire itself.

I was looking to know if there was the possibility of modifying any of the changes, either by extending, or otherwise, the weight of the query in another field other than _id since Elastisearch indexes usually contain, for convenience, some field, that serves the purpose of the arrangement, could be passed as a parameter.

But I didn't find anything about it, not even after reviewing the Livewire code.

Thanks anyway, Mrs. IA.

abkrim

2 years ago

Level 13

@LaryAI

The problem only affects Filament. I created a proof of concept with Livewire and it worked, because in Livewire I can do the query in a different way

public function render(): View
    {
        return view('livewire.work-analyzers.list-work-analyzers', [
            'workAnalyzers' => WorkAnalyzer::paginate(10),
        ]);
    }

This work fine.

Question, is: Is possible work with Filament and pdphilip/laravel-elasticsearchp?

Please or to participate in this conversation.