使用CURL检测坏链接引用者邻域

Question

I'm trying to use CURL to assess the visitors on my site. I'd like to see if they are being linked from a bad neighborhood or not. Most of the time my current code works, but not always.

I'm having a bit of trouble making my CURL able to fool all servers. How do I make my CURL headers totally convincing, and remove any possible clues that I'm using CURL?

<?php 

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"http://www.example.com");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$vars);  //Post Fields
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

$headers = array();
$headers[] = 'X-Apple-Tz: 0';
$headers[] = 'X-Apple-Store-Front: 143444,12';
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
$headers[] = 'Accept-Encoding: gzip, deflate';
$headers[] = 'Accept-Language: en-US,en;q=0.5';
$headers[] = 'Cache-Control: no-cache';
$headers[] = 'Content-Type: application/x-www-form-urlencoded; charset=utf-8';
$headers[] = 'Host: www.example.com';
$headers[] = 'Referer: http://www.example.com/index.php'; //Your referrer address
$headers[] = 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/28.0';
$headers[] = 'X-MicrosoftAjax: Delta=true';

curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$server_output = curl_exec ($ch);

print $server_output;

curl_close ($ch);    

if (strpos($output,'sex') !== false) 
{
    echo 'sex';
}

?>

For example, a certain well known adult video site with an orange logo that looks a lot like the YouTube logo (maybe you guys know the one) responded with this:

403 Forbidden

Request forbidden by administrative rules. 
__SERVERNAME__

Answer 1

In chrome dev tools, you can obtain the full HTTP request chrome used against a URL by:

• opening dev tools
• goto the "network" tab
• request the URL you want - if you are already on the target page - hit F5 or reload.
  Dev tools will then generate a list of HTTP requests (and responses) made
• right click on the HTTP request / URL you are interested in
• click the "save as curl" option and you will now have the full HTTP request details (for command line curl ) in your clipboard.

By using these values when sending an HTTP request your request will ostensibly appear to be made by a Chrome web browser.