查找页面上与页面位于同一域的链接

Question

I am building a crawler which starts from a webpage of website, say example.com, and find all the links on this page which are on same domain.

So suppose we have example.com/hello.php, facebook.com/hello.php on this page. So I only want to list www.example.com/hello.php.

I am using PHP Simple HTML DOM Parser(simplehtmldom.sourceforge.net/).

$html = file_get_html('http://www.example.com/');
// Find all links 
foreach($html->find('a') as $element) {
    $uri = $element->href;
    //Now how to check if $uri belongs to same domain?
}

Answer 1

Assuming, all your URLs are already absolute* URLs as in http://example.com/hello.php. Then you'll use parse_url to get the hosts of all your URLs.

php > $url = "http://example.com/hello.php";
php > print parse_url($url, PHP_URL_HOST);
example.com

You now just have to compare the host of the links to the host of the site you're currently crawling. If comparing the hosts is not enough, you have to extract the domains from the hosts. This is not easy, as there is no rule for it. https://www.publicsuffix.org/ has all the information you'll need for this task, though. This includes a PHP URL parser library.

*(URLs that are not absolute are of course on the same domain, so you don't need them for your decision.)