我如何通过PHP查找所有URL？

Question

I'm wondering if there's some code or library for getting all urls under a domain. I need to find all urls for a domain.

For example, if my domain is https://stackoverflow.com/ and I'd like to find all question url's like this:

1. [Java lib or app to convert CSV to XML file?
2. [https://stackoverflow.com/questions/456/what-can-i]
3. [https://stackoverflow.com/questions/789/where-can-i]

I don't know about how many questions are under the domain, but I have to create an engine for searching all the urls and then after finding the urls I need to insert the content into my database.

I will create a small search engine for my 5 web pages.

Can anyone help please?

Thanks,

Answer 1

Lucene search allows you to easily index your pages so they can be searched efficiently and accurately.

See Zend_Search_Lucene for a PHP implementation of Lucene serach.

You still have to spider your site and build the index which is another issue. You could use a software like Teleport Pro to spider your site and give you a list of URLs which you can then feed to a PHP script that gets the contents of all the pages and feeds them to Zend_Search_Lucene to build an index. You can also write the crawler in PHP or use an existing solution. A search for php crawler yields many things, including this useful php crawler.