How to open a 2.8 GB json file in php?

I have a 2.8 GB json file that I want to load . But directly loading to memory leads to memory exhaustion . So the idea may be to load the data in chunks or any other efficient way . What is the most efficient way of doing this? Code examples will be appreciated Thanks for the help

ahmeddabak

7 years ago

Level 47

salsify/jsonstreamingparser

This is a simple, streaming parser for processing large JSON documents. Use it for parsing very large JSON documents to avoid loading the entire thing into memory, which is how just about every other JSON parser for PHP works.

$stream = fopen('doc.json', 'r');
$listener = new YourListener();
try {
  $parser = new \JsonStreamingParser\Parser($stream, $listener);
  $parser->parse();
  fclose($stream);
} catch (Exception $e) {
  fclose($stream);
  throw $e;
}

halaxa

5 years ago

Level 1

Give a try to JSON Machine and simply iterate a JSON file or stream of any size with pure foreach. No other setup necessary.

$users = \JsonMachine\Items::fromFile('500MB-users.json');
foreach ($users as $id => $user) {
    // just process $user as usual
}

2 likes

Please or to participate in this conversation.