I have a Windows client application that consumes a php page hosted in a shared commercial webserver.
In this php page I am returning an encrypted json. Also in this page I have a piece of code to keep track of which IPs are visiting this php page, and I have noticed that there is a spyder/Nutch-2
crawler visiting this page.
I am wandering how is possible that a crawler could find a page that is not published in any search engines. I there a way to block crawlers from visiting this specific page?
Shall I use .htaccess
file to configure it?