浏览器模拟技巧/提示[关闭]

Question

I'm trying to simulate a web browser in order to log into a secure site, where the site's backend seems to be written in some mix of PHP and ASP.NET, and retrieve some user details.

In order to fit my own project, the simulation results (i.e. the user details) must be returned to a PHP script for processing.

So far I've been working with CURL in PHP to do this, and realised that the site is far too complicated to use CURL effectively, and this method is far too slow to develop. What I would like is some sort of browser simulator that can:

• Execute JavaScript
• Submit forms
• Click links
• Handles cookies
• Uses ASP.NET postbacks
• Can access the DOM

Basically something that behaves exactly like a real browser, and can return the page source to me.

I've explored the Snoopy class in PHP and Capybara in Ruby. If I don't get any better options I will be forced to implement with one of these.

Answer 1

You have two options:

1. Use a headless browser. This is basically browser without any graphical output, which can be controlled via. code. You can check out Selenium and PhantomJS, there probably exists bindings for your language of choice.

2. Reverse their site. Do the login flow and actions needed to get to the resource you need, and look at the network traffic, for example with Chrome's developer tools. Look at the requests, headers and form data needed for the endpoints in question and emulate that in the code.