Hi Everyone,
I'm working on an API endpoint that returns data from an Eloquent model, which contains around 400,000 records. While I've already implemented pagination, the requirement is to return the entire dataset in the JSON response when hitting the endpoint in Postman.
Given the size of the dataset, what are some best practices or optimization techniques I can use to handle this efficiently to avoid performance issues and memory overload?
Any advice or guidance would be greatly appreciated!
Thanks in advance!
You can perhaps have it as a downloadable file.
Just curious, why all the data instead of just what's needed.
Also would a web app work better instead of an API? Or even a multi tenant setup.
I see so many API's that a regular web app would be better suited.
In a database, the idea is to query what's needed.
@jlrdw The goal is to return a fresh set of records, as the table is regularly truncated and repopulated with new data.
This table consists of 14 columns, all of which need to be included in the response. Additionally, a third-party system will be consuming the entire dataset to populate their own system.
Given these requirements, I'm looking for the most efficient way to return the full dataset in a single JSON response. Any advice on optimizing this process would be helpful!
You can stream the data: https://laravel.com/docs/11.x/responses#streamed-json-responses
That'll massively reduce memory usage if you previously loaded the entire result set into memory before serialization.
But retrieving 400k records is a heavy operation. You should cache that somewhere instead of doing a DB query on every API call.
Please or to participate in this conversation.