cURL GET方法在php中失败，但浏览器和控制台成功完成

Question

I apologize if the title is inappropriate, but I kind of could not think of a better definition for it.

I am going nuts over this problem. I have been working on collecting feeds and data via cURL for the past 5+ years and have never encountered this kind of situation. I have a large json to collect over the GET method from a remote server via HTTPS from address that looks something like this https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1

someParam is changeable, and for some values with lower amount of data everything works fine, almost identical speeds to browser, but in several cases cURL always goes to tiomeout set, while in browser and from console everything works fine

PHP

My cURL is as follows:

$ch = curl_init();
$url = 'https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1';
curl_setopt($ch, CURLOPT_URL, $url);

curl_setopt($ch, CURLOPT_HEADER, 0);
// I've added this user agent as it is the same as the one Chrome uses
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36');

curl_setopt($ch, CURLOPT_POST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
// I have tried removing the SSL part below, but no difference
curl_setopt($ch, CURLOPT_SSL_CIPHER_LIST, "HIGH:!SSLv3s");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // tried this with true, but no difference
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT, 1200); // what ever the timeout I set the cURL always goes to timeout
curl_setopt($ch, CURLOPT_VERBOSE, true);

$response = curl_exec($ch);
if (curl_errno($ch)) {
    print("cURL error: " . curl_error($ch));
    print_r(curl_getinfo($ch));
} else {
    print_r(json_decode($response));
}
curl_close($ch);

This is the verbose output:

* Hostname was NOT found in DNS cache
*   Trying 12.34.567.89...
* Connected to private.example.com (12.34.567.89) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSL connection using ECDHE-RSA-AES256-SHA384
* Server certificate:
*    subject: OU=Domain Control Validated; CN=*.example.com
*    start date: 2016-03-03 09:41:38 GMT
*    expire date: 2018-03-04 09:52:18 GMT
*    subjectAltName: private.example.com matched
*    issuer: C=US; ST=Arizona; L=Scottsdale; O=Starfield Technologies, Inc.; OU=http://certs.starfieldtech.com/repository/; CN=Starfield Secure Certificate Authority - G2
*    SSL certificate verify ok.
> GET /thisDotNetEndPoint?token=bla-bla-trutj&someParam=1 HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36
Host: private.example.com
Accept: */*                                                                              */
* Operation timed out after 1200001 milliseconds with 0 bytes received
* Closing connection 0

It always goes to timout whatever the timout I set, tried even setting it to 2 hours.

I've even tried adding these but no difference:

curl_setopt($ch, CURLOPT_NOSIGNAL, 1);

curl_setopt($ch, CURLOPT_LOW_SPEED_LIMIT, 1);
curl_setopt($ch, CURLOPT_LOW_SPEED_TIME, 1200);

Browser

When I enter the same url in browser the response comes back in 6-9 minutes

cURL from console

I have used the simplest command and it works in same time as browser:

$ curl -X GET -v 'https://private.example.com/thisDotNetEndPoint?token=bla-bla-trutj&someParam=1'

Verbose output:

* Hostname was NOT found in DNS cache
*   Trying 12.34.567.89...
* Connected to private.example.com (12.34.567.89) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES256-SHA384
* Server certificate:
*    subject: OU=Domain Control Validated; CN=*.example.com
*    start date: 2016-03-03 09:41:38 GMT
*    expire date: 2018-03-04 09:52:18 GMT
*    subjectAltName: private.example.com matched
*    issuer: C=US; ST=Arizona; L=Scottsdale; O=Starfield Technologies, Inc.; OU=http://certs.starfieldtech.com/repository/; CN=Starfield Secure Certificate Authority - G2
*    SSL certificate verify ok.
> GET /thisDotNetEndPoint?token=bla-bla-trutj&someParam=1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: private.example.com
> Accept: */*                                                                              */
>
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Type: application/json; charset=utf-8
< Server: Microsoft-IIS/8.5
< X-StackifyID: V1|b8b10c35-2649-4f67-ba6a-b5ad15ef553b|C56050|CD18|
< Set-Cookie: .ASPXANONYMOUS=looI88UVBp6Cg5tLkzVejO4CNRilhyKjMY4hFqhuO48vdVT19U8h5oisC9khFv1rOmH6Ii_lEec-9XhipEvh1UkewhufqfmlTGFsyQCaML06NVa-5-Vr_OikZb07R6pdHCeRtn9liBVJfamJmXiElA2; expires=Thu, 02-Feb-2017 20:54:18 GMT; path=/; HttpOnly
< X-AspNetMvc-Version: 5.2
< Rx-CID: ae9907d6fc394b24b6599e74ab5a668f
< Rx_RequestId: f3fff82c4de04bba90b2bbc5704ac787
< X-Powered-By: ASP.NET
< Strict-Transport-Security: max-age=31536000
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Headers: rx-cid
< Date: Fri, 25 Nov 2016 10:25:00 GMT
< Content-Length: 2231472
<
[and the response is printed here]

Any ideas?

Thanks in advance.

Answer 1

Did you notice the difference between the console and your php verbose output? The useragent is missing in your php code. curl commandline by default adds this useragent, whereas the php-curl doesn't.

User-Agent: curl/7.35.0

Use the option CURLOPT_USERAGENT.

curl_setopt($ch, CURLOPT_USERAGENT, "Opera 11.0");