i have Problem with my Scraping Script. I like to select the value from my sql by id in a while and save the the scraped content in my database. its works but the curl script send a wrong result to my database, everytime the content from the first request.
My Code:
<?php
error_reporting(E_ALL);
include('db.php');
$i = 1;
while ($i <= 5)
{
$sql = "SELECT * FROM `plz` WHERE `id` = '$i'";
$row = mysql_fetch_assoc( mysql_query($sql) );
$plz = $row['plz'];
//create array of data to be posted
$post_data['adv_plz'] = "$plz";
$post_data['finda'] = 'adv';
$post_data['lang'] = 'de_DE';
//traverse array and prepare data for posting (key1=value1)
foreach ( $post_data as $key => $value) {
$post_items[] = $key . '=' . $value;
}
//create the final string to be posted using implode()
$post_string = implode ('&', $post_items);
//create cURL connection
$curl_connection = curl_init('https://www.domain.de/');
//set options
curl_setopt($curl_connection, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($curl_connection, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($curl_connection, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl_connection, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl_connection, CURLOPT_FOLLOWLOCATION, 1);
//set data to be posted
curl_setopt($curl_connection, CURLOPT_POSTFIELDS, $post_string);
//perform our request
$result = curl_exec($curl_connection);
//show information regarding the request
$result = utf8_encode($result);
mysql_query("UPDATE plz SET content = '$result' WHERE id = '$i'");
$i++;
}
?>
Here is the Content of the echo $post_string
adv_plz=01000&finda=adv&lang=de_DE
adv_plz=01000&finda=adv&lang=de_DE&adv_plz=01001&finda=adv&lang=de_DE
adv_plz=01000&finda=adv&lang=de_DE&adv_plz=01001&finda=adv&lang=de_DE&adv_plz=01002&finda=adv&lang=de_DE
adv_plz=01000&finda=adv&lang=de_DE&adv_plz=01001&finda=adv&lang=de_DE&adv_plz=01002&finda=adv&lang=de_DE&adv_plz=01003&finda=adv&lang=de_DE
adv_plz=01000&finda=adv&lang=de_DE&adv_plz=01001&finda=adv&lang=de_DE&adv_plz=01002&finda=adv&lang=de_DE&adv_plz=01003&finda=adv&lang=de_DE&adv_plz=01004&finda=adv&lang=de_DE
I hope somebody can help me. If i try the script not in a while it works perfectly.
</div>