如何实现网站的cookies scraper？

Question

I need to create a script that make an HTTP request and simulate browser's behavior when it comes to cookie management. That means that it has access to all the cookies set by that 'page' (So server-set cookies, async client-set cookies, cookies of every domain.)

I mean, inspecting the page using a console, i can see all the cookies from all domains on that page.

I am figuring out a way to do that from my code. I am almost sure i have to somehow run page's javascript and to emulate the DOM behavior (thought to use jsDom in a node.js script).

But, but. Still have many doubts. Any suggestion on how i can intercept all cookies by simulating browser behavior ?

Any suggestion about how to implement it would be really appreciated.

for those who have time
I had a weird idea about how to implement it: Would it make sense to overload the prototype of the function of the xhr object that handles HTTP response (i mean making something like: this), to check all the client-loaded cookies ?
I need it to work on any page, even those which don't use the native XMLHttpRequest object. Any suggestion around this ?

Answer 1

Browsers just send HTTP requests to the server and you don't necessarily need a DOM loaded. If you can reverse engineer the requests that you want to send, you can easily mock the behavior of a web page or workflow. When you send a request with cURL from PHP, you'll need to store the cookies in a cookie jar to maintain cookies across requests. Something like this should get you started:

function load($url, $postData = array())
{
    $useragent = "User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.109 Safari/537.36";

    $curl = curl_init();
    curl_setopt($curl, CURLOPT_URL, $url);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
    curl_setopt($curl, CURLOPT_HEADER, FALSE);
    curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
    curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
    curl_setopt($curl, CURLOPT_ENCODING, 'UTF-8');
    curl_setopt($curl, CURLOPT_USERAGENT, $useragent);
    curl_setopt($curl, CURLOPT_POST, !empty($postData));
    if(!empty($postData)) curl_setopt($curl, CURLOPT_POSTFIELDS, $postData);
    curl_setopt($curl, CURLOPT_COOKIEFILE, $cookieFile);
    curl_setopt($curl, CURLOPT_COOKIEJAR, $cookieFile);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
    $page = curl_exec ($curl);
    curl_close ($curl);

    return $page;
}

If you want to run a headless browser that maintains cookies in a cookie jar, I would recommend something like PhantomJS. You can then load pages and execute code within a page's context:

"use strict";
var page = require('webpage').create();

page.onConsoleMessage = function(msg) {
    console.log(msg);
};

page.open("http://phantomjs.org/", function(status) {
    if (status === "success") {
        page.includeJs("http://ajax.googleapis.com/ajax/libs/jquery/1.6.1/jquery.min.js", function() {
            page.evaluate(function() {
                console.log("$(\".explanation\").text() -> " + $(".explanation").text());
            });
            phantom.exit(0);
        });
    } else {
      phantom.exit(1);
    }
});

The code can be found on Github: https://github.com/ariya/phantomjs/blob/master/examples/phantomwebintro.js