防止机器人刮擦我们的内容/超载我们的服务器

Question

We are planning to offer a job platform-service for some firms. We already have a few thousand jobs that we could offer to all our guest/visitors.

Since yesterday we noticed that our server-load is crazy and when we checked the logs we saw that we had multiple site-request per second from different IP addresses. However the order in which the pages were called indicate it was the same user / bot

We want to be available for the public but if bots are slowing our server massively down or forcing us for new hardware then we are in trouble.

We are currently displaying all our job-content in iframes, would an encoder like: http://www.tareeinternet.com/scripts/iframe-encoder/

help to solve our problem?

Or what options do we have? Its especially annoying since we don't have user-sessions or recurring IP-Addreses (I think they are using proxys that switch regulary)

Answer 1

Have you checked the headers for recurring data? If they, for example, have a recurring user-agent you can can block those:

• Apache:

SetEnvIfNoCase User-Agent "^Wget" bad_bot
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "^EmailWolf" bad_bot
<Directory "/var/www">
        Order Allow,Deny
        Allow from all
        Deny from env=bad_bot
</Directory>

• Code: You can check each requests for that specific header and redirect it to somewhere else.

</div>