脚本执行后通过PHP获取html页面

Question

I did a lot of search... tried some methods.. There is webpage that html contents come after script execution.

I used phantomJS with different method.

1-) Checking with document.ready

var page = require('webpage').create();
console.log('The default user agent is ' + page.settings.userAgent);
page.settings.userAgent = 'SpecialAgent';
page.open('http://sosyal.hurriyet.com.tr/yazar/niobe_141/seni-unutmuyoruz-pasam_40011882', function(status) {
 function checkReadyState() {
        setTimeout(function () {
            var readyState = page.evaluate(function () {
                return document.readyState;
            });

            if ("complete" === readyState) {
                onPageReady();
            } else {
                checkReadyState();
            }
        });
    }

    checkReadyState();
});

function onPageReady() {
    var htmlContent = page.evaluate(function () {
       return document.body.textContent;
    });

    console.log(htmlContent);

    phantom.exit();
}

Result:Script not loaded so unloaded html returned..

2-)Setting timeout too long

  page.open(address, function (status) {
    if (status !== 'success') {
        console.log('Unable to load the address!');
        phantom.exit();
    } else {
        window.setTimeout(function () {
            var htmlContent = page.evaluate(function () {
       return document.getElementsByClassName('hsaalicc-text').textContent;
    });

    console.log(htmlContent);

        }, 1000); // Change timeout as required to allow sufficient time 
    }
});

Result:Script not loaded so unloaded html returned..

So although I'm android developper and have not too much jquery knowlodge looked page code with chrome developper console... And I see all data that should be load is in script with window.articleDetailData

Moreover I found the function that load data content.

('#templateArticleDetail').tmpl(data).appendTo('#articleDetailContainer');

There is no time parameter,but in mobile device it takes time. But in code I understand when page loaded it should copy to #articleDetailContainer

So my question 1-) why document ready and high timeout not return loaded script page with phantomJS 2-) Is there a way to parse windows.data under script tag??

If I could not find any easy way,will use regex to parse script

Answer 1

You can use PhantomJsCloud.com to do this, though in the answers below, I'll try to explain the process if you want to try to use your own phantomjs.exe instance. (Disclosure: I wrote PhantomJsCloud)

Docs for PhantomJsCloud here: http://api.phantomjscloud.com/

1) Waiting for AJAX?

Using PhantomJsCloud: Just a normal request, everything is automatically loaded properly. Here is your page rendered as a PNG: http://api.phantomjscloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/?request={url:%22http://sosyal.hurriyet.com.tr/yazar/niobe_141/seni-unutmuyoruz-pasam_40011882%22,renderType:%22png%22}

Using PhantomJs.exe: If you do it yourself, you need to make sure all ajax resource requests are complete before rendering. (see the WebPage.OnResourceRecieved() api)

2) Parse windows.data?

Using PhantomJsCloud: Set pageRequest.requestType="script", Use pageRequest.scripts to execute your script, and return the data you want. For example: http://api.phantomjscloud.com/api/browser/v2/a-demo-key-with-low-quota-per-ip-address/?request={url:%22http://example.com%22,renderType:%22script%22,scripts:{loadFinished:[%22return%20{hello:%27world%27,host:document.location.host};%22]}}

Using PhantomJs.exe: You need to use http://phantomjs.org/api/webpage/method/include-js.html or injectJs and load a script that will do your parsing, and then send it back to your phantomjs.exe code via http://phantomjs.org/api/webpage/handler/on-callback.html