I am building a website which scrapes data from another website,stores it in a database and shows it in the form of a table. Everything works fine as long as the number of rows are less (around 100), but when the data set increases, say 300 rows or more the data gets stored in the database (phpmyadmin) but nothing shows on the screen and the site just keeps loading. Below is a section of the php script i am running:
<?php
// configuration
require("../includes/helpers.php");
// initializing current page and number of pages
$page = 0;
$pages = 1;
// scrape data from each page
while($pages--)
{
// next page
$page++;
// scrape data from shiksha.com
$string = @file_get_contents("http://www.shiksha.com/b-tech/colleges/b-tech-colleges-".urlencode($_POST["city"])."-{$page}");
if($string === false)
apologize("Please enter a valid city name");
if($page === 1)
{
// counting total number of pages
preg_match_all('/class=" linkpagination">/',$string,$result);
$pages = sizeof($result[0]);
}
// passing the string for scraping data and storing in database
get_college_info($string,$page);
// delay for 2s
sleep(2);
}
// querying the infrastructure table for facilities of all colleges
$infra = query("SELECT college_id,facilities FROM infrastructure ");
// preparing query and selecting data from table college_info
$result = query("SELECT * FROM college_info");
// render(output) results
render("result.php",["title" => "result","infra" => $infra,"result" => $result]);
}
}?>
interestingly, if i already have the data stored in my db and I just retrieve and print it , everything works fine and all the data ,however large it is,gets printed. I have no clue whats the problem. PS : I have already tried set_time_limit().
