你如何提交一个不会立即使用Python返回结果的PHP表单？

Question

There is a PHP form which queries a massive database. The URL for the form is https://db.slickbox.net/venues.php. It takes up to 10 minutes after the form is sent for results to be returned, and the results are returned inline on the same page. I've tried using Requests, URLLib2, LXML, and Selenium but I cannot come up with a solution using any of these libraries. Does anyone know of a way to retrieve the page source of the results after submitting this form?

If you know of a solution for this, for the sake of testing just fill out the name field ("vname") with the name of any store/gas station that comes to mind. Ultimately, I need to also set the checkboxes with the "checked" attribute but that's a subsequent goal after I get this working. Thank you!

Answer 1

I usually rely on Curl to do these kind of thing. Instead of sending the form with the button to retrieve the source, call directly the response page (giving it your request). As i work under PHP, it's quite easy to do this. With python, you will need pycURL to manage the same thing.

So the only thing to do is to call venues.php with the good arguments values thrown using POST method with Curl.

This way, you will need to prepare your request (country code, cat name), but you won't need to check the checkbox nor load the website page on your browser.

set_ini(max_execution_time,1200) // wait 20 minutes before quitting
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch, CURLOPT_URL, "https://db.slickbox.net/venues.php");
curl_setopt($ch, CURLOPT_HEADER, 0);

// prepare arguments for the form
$data = array('adlock   ' => 1, 'age' => 0,'country' => 145,'imgcnt'=>0, 'lock'=>0,'regex'=>1,'submit'=>'Search','vname'=>'test');

//add arguments to our request
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
//launch request
if( ! $result = curl_exec($ch))
{
    trigger_error(curl_error($ch));
} 
echo $result;