如何阻止spyder / Nutch-2等爬虫访问特定页面？

Question

I have a Windows client application that consumes a php page hosted in a shared commercial webserver.

In this php page I am returning an encrypted json. Also in this page I have a piece of code to keep track of which IPs are visiting this php page, and I have noticed that there is a spyder/Nutch-2 crawler visiting this page.

I am wandering how is possible that a crawler could find a page that is not published in any search engines. I there a way to block crawlers from visiting this specific page?

Shall I use .htaccess file to configure it?

Answer 1

You can indeed use a .htaccess. robots.txt is another option but some crawlers will ignore this. You can also block specific user agent strings. (They differ from crawler to crawler)

robots.txt:

User-agent: *
Disallow: /

This example tells all robots to stay out of the website: You can block specific directories

Disallow: /demo/

More information about robots.txt