为什么curlExec（）的访问权限少于浏览器？

Question

I have an application that downloads favicon files. Recently I switched from using file_get_contents to curlExec because it has a higher success rate.

However when I try and download from www.prisonexp.org , instead of sending the actual file data they send the text Forbidden. Normally, I would see a bunch of binary data converted to ASCII in this field in the test script.

I find this strange because I can just browse to the file in the browser and download it manually.

Is this valid? Or am I missing something? How are they preventing a download one way but not another. To see the test script in action go here.

Test Script

As a side, how can I detect when instead of sending binary data, a server sends a message as text? I could just check for "Forbidden" but I'm not sure if this is a standard response.

Research / Update

Download Methods

Answer 1

Things that can be done programmatically have to be throttled back or servers could become overwhelmed.

Reading the user-agent is one way this is done.

Wget might have more access as it is commonly used as a command line tool. Similarly images delivered to your client you will have access to directly.