Store data from several pages of an API into database

I'm calling an API that has hundreds of records paginated with 20 results per page. I'm wanting to store the returned data into a table in my DB. The issue I face is I'm unsure how to move onto the next page and loop through those results.

There is the skip=0 parameter that I can adjust to switching pages, I'm just unsure on how to dynamically do this and keep looping through the results.

Here's my call which gets the first 20 and stores them

public function store(Request $request)
{
    $getData = Http::get('https://example-url.com/Search?House=1&IsCurrentMember=true&skip=0&take=20');
    $jsonResponse = $getData->json();

    foreach ($jsonResponse['items'] as $mp) {

        $mp = new Mp([
            'id' => $mp['value']['id'],
            'name' => $mp['value']['nameDisplayAs'],
            'party' => $mp['value']['latestParty']['name'],
            'constituency' => $mp['value']['latestHouseMembership']['membershipFrom'],
            'county' => '',
            'region' => '',
            'country' => '',
            'active' => $mp['value']['latestHouseMembership']['membershipStatus']['statusIsActive'],
            'image' => $mp['value']['thumbnailUrl']

        ]);

        $mp->save();
    }

    return view('admin.mps.index')->with('success', 'data was successfully add.');
}

As well as the data been returned via the API it also has the next and prev urls

"totalResults": 650,
"resultContext": "",
"skip": 0,
"take": 20,
"links": [
    {
        "rel": "self",
        "href": "/Members/Search?skip=0&take=20",
        "method": "GET"
    },
    {
        "rel": "page.next",
        "href": "/Members/Search?skip=20&take=20",
        "method": "GET"
    },
    {
        "rel": "page.previous",
        "href": "/Members/Search?skip=0&take=20",
         "method": "GET"
    }
]

SilenceBringer

4 years ago

Best Answer

Level 55

@cib88 something like

public function store(Request $request)
{
    $skip = 0;
    $take = 20;

    do {
        $getData = Http::get('https://example-url.com/Search?House=1&IsCurrentMember=true&skip=' . $skip . '&take=' . $take);
        $jsonResponse = $getData->json();

        // save

        $skip += $take;
    } while ($jsonResponse['totalResults'] > $skip);

    return view('admin.mps.index')->with('success', 'data was successfully add.');
}

by the way, I recommen to re-write storing too to insert data at once using DB facade (if you don't watch for model events)

Please or to participate in this conversation.