How to manage large amount of data in laravel API ?

I'm working on a project where I need to send any data through API Route like

Route::get('/allData','Controller@allData');

function Controller(){
$data=Data::get();
 return response()->json([
                 'data' => $data,
         ]);
}

There will a lot of data like 10 lakh 20 lakh something. now I am using that API route in another simple External HTML project to show table & chart. I think due to a load of data my page is not responding properly even I can't check to inspect. So is there any way to handle data properly and help to respond to my HTML page properly?

rodrigo.pedra

5 years ago

Level 56

How big is 10 lakh or 20 lakh? I am not familiar with this.

Anyway, most browsers will start having a hardtime rendering thousands of elements. Actually it really depends on how much CSS is involved, how deep is the DOM tree, if there are any JavaScripts hooks such as event listeners, etc. But as a rule of thumb consider a document with more than 10,000 elements a very hard one, specially in more underpowered devices such as mobile phones .

References:

https://stackoverflow.com/questions/2923524/how-many-divs-can-you-have-before-the-dom-slows-and-becomes-unstable

https://developers.google.com/maps/documentation/javascript/overview (see the embedded video)

So if 10 lakh is greater than that I would suggest that you consider paginating your data.

https://laravel.com/docs/8.x/pagination#introduction

Also retrieving large chunks of data from an API endpoint can potentially put a heavy load on your database if there many concurrent requests to that endpoint.

Consider caching or saving this large dataset to a static file and serve that. You can have job or scheduled tasks that update the cache/static file from time to time.

saurav77

5 years ago

Level 2

@rodrigo.pedra something like 1 million data maybe. I am using these data for filters like address filter, state filter, location filter, company filter, year filter, etc. so don't know why my page is not responding.it does not respond properly and cannot be clicked any button or neither right button mouse My code line is around like 750 only

rodrigo.pedra

5 years ago

Level 56

One million data records sent from the server to be processed by JavaScript or converted to DOM elements is a lot.

If you need to fetch data from server for dynamic filters, or dynamic search results, consider sending your filter, or search, criteria and returning a smaller subset of the results.

Take a look at Laravel Scout (for searching) https://laravel.com/docs/8.x/scout

Or an approach like Spatie's Laravel Query Builder https://github.com/spatie/laravel-query-builder for using parameters to apply filtering and sorting.

On the frontend side, to learn on how to send dynamic queries to the server and update the UI, read about, or watch tutorials about lookahead searches, data tables, and other techiniques.

This days this can be done with LiveWire (there is a ongoing series about it on Laracasts) or, more traditionally with some dynamic JavaScript with or without a framework such as Vue, React, and others.

Some references on this:

https://laracasts.com/series/livewire-basics

https://laracasts.com/series/lets-build-a-forum-with-laravel/episodes/94

https://laracasts.com/series/lets-build-a-forum-with-laravel/episodes/95

https://laracasts.com/series/lets-build-a-forum-with-laravel/episodes/97

https://laracasts.com/series/build-a-video-game-aggregator/episodes/16

https://laracasts.com/series/build-a-video-game-aggregator/episodes/17

https://laracasts.com/series/whatcha-working-on/episodes/16 (uses algolia, which is a paid service)

https://laracasts.com/series/guest-spotlight/episodes/3 (from Caleb Porzio, creator of Livewire)

saurav77

5 years ago

Level 2

@rodrigo.pedra thanks a lot for responding Okay, let's say I have only a link to the API route given by the laravel developer. I have a simple page like index.html page to draw a chart and an advanced table and fetch data like

 fetch('http://nextaustami.com.np/api/barChart')

Now My job is to plot a chart from those laravel API data and also keep data like company name, street, district, etc in a select box for filter and send request coming from the filter to the laravel function.Its simple HTML page only data are taken from another project so So what do you suggest? & thank you for responding. I hope you understand the problem

rodrigo.pedra

5 years ago

Best Answer

Level 56

Hi @saurav77 , sorry for the late response, I was busy this week.

Avoid converting this one million data points into DOM elements.
To download a file that big use a async approach, the easiest way is to use JavaScript's builtin fetch instead of regular XMLHttpRequest as the later will block your main thread until the download is finished, so the UI becomes unresponsive. You can experiment with a JavaScript Worker too.
When rendering a chart with so many points, consider down-sampling the dataset. There is no reason to plot 1 million data points in a chart that is a few pixels wide. Some chart libraries handle that automatically, at my head Hihgcharts and Fuisoncharts can handle that by down-sampling the dataset automatically for you. But those are paid libraries. If you need/want to use a free chart library you might have to down-sample the dataset yourself. Search NPM for a chart.js plugin named chartjs-plugin-downsample. Even if you are not using chart.js search this plugin codebase at GitHub for a function called downsample, it is a self contained function where you can learn how down-sampling works.
Regarding displaying the data as a searchable and filterable table, consider paginating the data in JavaScript, and only show a small subset of it. Rendering a table row for each 1 million records from your dataset will render the browser unresponsive. Some datatable slibraries already do that for you automatically. Most well known one is from DataTables.net. Although the example in their homepage is on top of an existing HTML table, you can use a JavaScript datasource instead. Search in their site's example section for examples on how to do it. If you are using a JavaScript framework you can Google for vue datatable, react datatable and so on. There is no shortage of datatables implementations out there for ready use or to learn on top of them.
Last, but not least, if your app is a separate Laravel app, you can interact with the external API server-side, cache the results (in a database table, for example) and use regular Laravel techiniques around it. Or even use LiveWire that excels on handlings this kind of UI (dynamic datatables), there is even a Laracast lesson about that. To keep your cached version in sync you can have a scheduled command that downloads a new copy of the external API data from time to time and update your local copy.

Hope some of these tips help anyhow. Have a nice day =)

Please or to participate in this conversation.