I have been trying to figure out how to run a python web scraper from a PHP interface. The web scraper puts data into a mysql database, and the PHP interface echos the data in a table. I have spent a lot of time researching this, however the data never shows up in the table, even when I keep the data in the database, and reload the interface, no data shows up. Also When i call the python script in the terminal and then open the interface, the data arrives. Is there any way people could recommend doing this? the first php script is as follows:
<!DOCTYPE HTML>
<html>
<body>
<?php
echo "3h test";
exec("python 3h.py");
?>
<form>
<input type="button" value="ViewCode" onclick="location="'python 3h.py'>
</form>
<a href="action.php">Run script.</a>
</body>
</html>
the python crawler is:
from urllib import urlopen
from bs4 import BeautifulSoup
import pymysql.cursors
# Webpage connection
page1 = urlopen("http://www.officialcharts.com/charts/singles- chart/19800203/7501/")
page2= urlopen("http://www.officialcharts.com/charts/")
# Grab title-artist classes and store in recordList
bsObj0 = BeautifulSoup(page1,"html.parser")
bsObj1 = BeautifulSoup(page2,"html.parser")
recordList0 = bsObj0.findAll("div", {"class" : "title-artist",})
recordList = recordList0 + bsObj1.findAll("div", {"class" : "title-artist",})
connection = pymysql.connect(host='localhost',
user='root',
password='******',
db='crawler database',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
try:
with connection.cursor() as cursor:
for record in recordList:
title = record.find("div", {"class": "title",}).get_text().strip()
artist = record.find("div", {"class": "artist"}).get_text().strip()
sql = "INSERT INTO `mytable` (`title`,`artist`) VALUES (%s,%s)"
cursor.execute(sql,(artist,title))
connection.commit()
finally:
connection.close()
the python script is named 3h.py
the final php script is to long for me to bother typing out, this is the core:
<?php
$no = 1;
$total = 0;
while ($row = mysqli_fetch_array($query))
{
echo '<tr>
<td>'.$no.'</td>
<td>'.$row['itemName'].'</td>
<td>'.$row['itemCost'].'</td>
</tr>';
$no++;
}?>
