递归函数，直到完成值为真，并将每个响应合并为一个大的响应

Question

I have this function:

public function syncTerritoriesSOQLQuery($veevaToken, $instanceUrl, $tokenUrl)
{
    $soqlQuery2 = "SELECT Id,Name,LastModifiedDate FROM Territory";
    $soqlUrl2 = $instanceUrl.'/services/data/v28.0/query/?q='.urlencode($soqlQuery2);

    $curl = curl_init($soqlUrl2);

    curl_setopt($curl, CURLOPT_HEADER, false);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_HTTPHEADER, array("Authorization: OAuth $veevaToken"));

    $jsonResponse = curl_exec($curl);

    $status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
    if ($status !== 200) {
        $respObj['error'] = "Error: call to token URL $tokenUrl failed with status $status, response $jsonResponse, curl_error ".curl_error(
                $curl
            ).", curl_errno ".curl_errno($curl);

        return $respObj;
    }

    curl_close($curl);

    $soqlObj2 = json_decode($jsonResponse, true);

    return $soqlObj2;
}

When I call it I got this response:

{
  "totalSize": 6911,
  "done": false,
  "nextRecordsUrl": "/services/data/v28.0/query/01g8000002eI8dMAAS-2000",
  "records": [ ... ]
}

That means complete set of records will have 6911 items but on first request just the first 2000 are return since it's paginated. So I need to call the same code again and again and again until "done": true. In every new request I need to change the $soqlUrl2 by append the value return on each call. For example:

1st time:
$soqlUrl2 = $instanceUrl.'/services/data/v28.0/query/?q='.urlencode($soqlQuery2);

2nd time:
$soqlUrl2 = $instanceUrl."/services/data/v28.0/query/01g8000002eI8dMAAS-2000";    

3rd time:
$soqlUrl2 = $instanceUrl."/services/data/v28.0/query/01g8000002eI8dMAAS-4000";

...

That until done gets true on the response. I should also merge the whole JSON result from each iteration into a big one, lets said: result from 2nd time should be merged with result from first call meaning $soqlObj2 result from 3d time should be merge with the previous merge in pseudo-code:

merge($soqlObj2, 1st, 2nd, 3rd, ...)

I have been working on this function:

public function performPaginateSOQLQuery($veevaToken, $instanceUrl, $tokenUrl, $nextRecordsUrl, &$dataToSave = array()) {
    if ($nextRecordsUrl !== null && $nextRecordsUrl !== "") {
        $keepRequesting = true;

        while($keepRequesting){
            $nextRecordsUrlSOQLQuery = $instanceUrl.$nextRecordsUrl;

            $curl = curl_init($nextRecordsUrlSOQLQuery);

            curl_setopt($curl, CURLOPT_HEADER, false);
            curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
            curl_setopt($curl, CURLOPT_HTTPHEADER, array("Authorization: OAuth $veevaToken"));

            $jsonResponse = curl_exec($curl);

            $status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
            if ($status != 200) {
                $respObj['error'] = "Error: call to token URL $tokenUrl failed with status $status, response $jsonResponse, curl_error ".curl_error(
                        $curl
                    ).", curl_errno ".curl_errno($curl);

                return $respObj;
            }

            curl_close($curl);

            $nextRecordsUrlObj = json_decode($jsonResponse, true);
            $dataToSave[] = $nextRecordsUrlObj;

            echo "iteration 2", "
";
            echo "url: ", $nextRecordsUrl, "
";
            echo "done: ";
            var_dump($nextRecordsUrlObj['done']);
            echo "
";

            if($nextRecordsUrlObj['done'] === true) {
                $keepRequesting = false;
            }
        }

        return array('url' => $nextRecordsUrlObj, 'data' => $dataToSave);
    }
}

And I call it from the function above syncTerritoriesSOQLQuery() as follow:

$entireSoqlObj2 = $this->performPaginateSOQLQuery(
        $veevaToken,
        $instanceUrl,
        $tokenUrl,
        $soqlObj2['nextRecordsUrl']
);

But it is not working since I am getting this result:

iteration 0
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

iteration 2
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

iteration 2
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

iteration 2
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

iteration 2
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

Right output should be something like:

// 1st
url: /services/data/v28.0/query/01g8000002eYb8LAAS-2000
done: bool(false)

// 2st
url: /services/data/v28.0/query/01g8000002eYb8LAAS-4000
done: bool(false)

// 3rd
url: /services/data/v28.0/query/01g8000002eYb8LAAS-6000
done: bool(true)

Because totalSize is 6911, can any help me to fix my code? Where is my mistake?

NOTE: I ask this earlier today but is not clear to me at all so I've open a new question with more info.

Answer 1

You should have $soqlObj2['nextRecordsUrl'] inside your recursive function , which should be updated with $nextRecordsUrl every time. Now it is only checking the first page always and hence you are getting the url for second pages.

Also, I don't understand why you have so much code in the performPaginateSOQLQuery() , you just want to iterate over syncTerritoriesSOQLQuery() until done, right?

---

Update:

Updating your syncTerritoriesSOQLQuery() to loop over all pages. Added the comments everywhere I made the changes. Hope that helps.

<?php
public function syncTerritoriesSOQLQuery($veevaToken, $instanceUrl, $tokenUrl)
{
    $soqlQuery2 = "SELECT Id,Name,LastModifiedDate FROM Territory";

    // removed $instanceUrl, we'll add that inside while loop
    $instancePath = '/services/data/v28.0/query/?q='.urlencode($soqlQuery2);

    $final_output = array(); // all output will be appended to this array
    $done = False; // we set this variable to true, once `done` is true

    while( !$done ) {
        $soqlUrl2 = $instanceUrl . $instancePath;

        $curl = curl_init($soqlUrl2);

        curl_setopt($curl, CURLOPT_HEADER, false);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
        curl_setopt($curl, CURLOPT_HTTPHEADER, array("Authorization: OAuth $veevaToken"));

        $jsonResponse = curl_exec($curl);

        $status = curl_getinfo($curl, CURLINFO_HTTP_CODE);
        if ($status !== 200) {
            $respObj['error'] = "Error: call to token URL $tokenUrl failed with status $status, response $jsonResponse, curl_error ".curl_error(
                    $curl
                ).", curl_errno ".curl_errno($curl);

            return $respObj;
        }

        curl_close($curl);

        $soqlObj2 = json_decode($jsonResponse, true);

        // here, we change the $instancePath to nextRecordsUrl
        $instancePath = $soqlObj2['nextRecordsUrl'];

        //  set $done variable
        $done = $soqlObj2['done'];

        // appending the data obtained to $final_output array, you can change this string if u need.
        $final_output[] = $soqlObj2;
       }

    return $final_output;
}