从不同的域PHP加载网站

Question

In my application I am loading product info from a supplier:

$start_url = "http://www.example.com/product/product_code";

These URLs are usually redirected by the supplier's website, and I have written a function that successfully finds the destination URL, like so:

$end_url = destination( $start_url );
echo "<a href=\"$start_url\">start url</a>"; // link get redirected to correct page
echo "<a href=\"$end_url\">end url</a>"; // links straight to correct page, no redirection

However, if I want to get the HTML from the page...

echo file_get_contents( $start_url );  // 404
echo file_get_contents( $end_url );  // 404

...I just get the supplier's 404 page (not a generic one but their custom one).

I have allow_url_fopen enabled; file_get_contents( "http://www.example.com/" ) works fine.

I can use either URL to load the expected content in an iframe client-side, but XSS security prevents me extracting the data I need.

The only thing I can think of is if the site is using an URL rewriter, could this mess things up?

The PHP is running on my local machine, so it should appear no different from me looking at the website via a browser as far as I'm aware.

Answer 1

Thanks to @Loz Cherone ツ's comments, using cURL and changing the user agent worked.

$user_agent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13";

$url = $_REQUEST["url"];  // e.g. www.example.com/product/ABC123            

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);  // follows any redirection
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

echo curl_exec($ch);

curl_close($ch);

I then put the response into the srcdoc attribute of an iframe client-side so I can access the DOM.