Elasticsearch Limit in PHP

Hi, I am new on elastivsearch. I have a block code for take value from elasticsearch. Normally, max limit 10.000 on elasticsearch, however I can't take 10.000, I take 7.022. If I try take 7.023 value it gives me error. Error is message: "Undefined index: area_1". If I delete area_1 this time gives error from area_2. Here my code:

use Elasticsearch\ClientBuilder;
use DataTables;

public function index()
{
    $hosts = ['localhost:9200'];
    $client = ClientBuilder::create()->setHosts($hosts)->build();

    $params = [
        'index' => 'data',
        'type' => 'long',
        'body' => [
            'size' => 7023, // limit the number of results, It gives error If 7023, no error 7022
            'query' => [
                'match_all' => new \stdClass(),
            ],
        ],
    ];

    $results = $client->search($params);

    $data = [];
    foreach ($results['hits']['hits'] as $hit) {
        $source = $hit['_source'];
        $data[] = [
            'id' => $hit['_id'],
            'area_1' => $source['area_1'],
            'area_2' => $source['area_2'],
            // repeat for all 25 areas
        ];
    }

    return DataTables::of($data)->make(true);
}

How can I fix this problem? I have 14.598 data on my elasticsearch index.

LaryAI

3 years ago

Level 58

It looks like you are running into a memory issue. The error you are getting is likely due to the fact that you are trying to load too much data into memory at once. To fix this, you can try increasing the memory limit in your php.ini file. You can also try breaking up your query into smaller chunks and looping through them. For example:

use Elasticsearch\ClientBuilder;
use DataTables;

public function index()
{
    $hosts = ['localhost:9200'];
    $client = ClientBuilder::create()-&gt;setHosts($hosts)-&gt;build();

    $data = [];
    $offset = 0;
    $limit = 1000;
    while (true) {
        $params = [
            'index' =&gt; 'data',
            'type' =&gt; 'long',
            'body' =&gt; [
                'from' => $offset,
                'size' =&gt; $limit,
                'query' =&gt; [
                    'match_all' =&gt; new \stdClass(),
                ],
            ],
        ];

        $results = $client-&gt;search($params);

        foreach ($results['hits']['hits'] as $hit) {
            $source = $hit['_source'];
            $data[] = [
                'id' =&gt; $hit['_id'],
                'area_1' =&gt; $source['area_1'],
                'area_2' =&gt; $source['area_2'],
                // repeat for all 25 areas
            ];
        }

        if (count($results['hits']['hits']) < $limit) {
            break;
        }

        $offset += $limit;
    }

    return DataTables::of($data)-&gt;make(true);
}

This code will loop through the results in chunks of 1000, so you don't have to load all 14,598 records into memory at once.

tisuchi

3 years ago

Level 70

@onurzdgn It seems that the memory limit issue.

Either you use a chunk of data instead of all or increase the memory limit in php (if possible).

onurzdgn

3 years ago

Level 2

@tisuchi I try increase the limit but still sam problem.

tisuchi

3 years ago

Level 70

@onurzdgn What if you set the starting offset by from?

e.g.

$params = [
    'index' => 'data',
    'type' => 'long',
    'body' => [
        'size' => 7000,
        'from' => 7022,
        'query' => [
            'match_all' => new \stdClass(),
        ],
    ],
];

onurzdgn

3 years ago

Best Answer

Level 2

@tisuchi Thank you again, problem is not from code. Problem in the elasticsearch data was different :D but I fixed thank you again.

Please or to participate in this conversation.