robots.txt禁止空白的PHP页面并包括？ [关闭]

Question

Should I disallow certain PHP pages in robots.txt that aren't meant to display content to the user, but rather run a script and return blank or output very little code that is meant to be used else where?

Examples:

• PHP pages that turn up blank. Example is a PHP page to send email
• PHP pages that are called by Ajax and output nothing.
• PHP pages that are called by Ajax and output response code such as JSON or XML.
• Include pages such as templates for your page's header or a config file.
• PHP pages which simply redirect to another page.

I appreciate your guys' help.

Answer 1

These are excellent examples of content that search engines don't want or need to crawl and index. You should block these with your robots.txt file. Especially if crawling them would cause actions to occur that is only intended for human consumption.

FYI, includes and config files should not be in your webroot and thus not accessible to crawlers or users via the web. If they are you should consider moving them.