为什么更改用户代理时会被拒绝？

Question

I have a simple request to get the robots.txt file from nike.com. A normal GET request in Go returns the file as expected. In my current browser (Chrome) I can access the file fine. However, when I try to change my UA in Go to the same as that of my browser, I get a 403 error.

In other words, I can access the page using the default settings in Go, proving that my code is correct, but once I change the UA I get a 403 (even though that same UA in my browser works).

If I change the UA to something random (e.g. "Not me") then I get a 200. Why can't I use my browser's UA?

package main

import (
    "log"
    "net/http"

    "github.com/davecgh/go-spew/spew"
)

func main() {
    var resp *http.Response
    var err error
    u := "http://www.nike.com/robots.txt"

    for _, ua := range []bool{false, true} {
        client := &http.Client{}
        req, _ := http.NewRequest("GET", u, nil)
        if ua == true {
            req.Header.Set("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36")
        }
        resp, err = client.Do(req)
        if err != nil {
            log.Fatalln(err)
        }
        log.Println("StatusCode", resp.StatusCode)
        spew.Dump(resp.Request)
        log.Println("----------------------------")
    }

}

The response:

2016/06/22 16:56:57 StatusCode 200
(*http.Request)(0xc8200d02a0)({
 Method: (string) (len=3) "GET",
 URL: (*url.URL)(0xc820076280)(http://www.nike.com/robots.txt),
 Proto: (string) (len=8) "HTTP/1.1",
 ProtoMajor: (int) 1,
 ProtoMinor: (int) 1,
 Header: (http.Header) {
 },
 Body: (io.ReadCloser) <nil>,
 ContentLength: (int64) 0,
 TransferEncoding: ([]string) <nil>,
 Close: (bool) false,
 Host: (string) (len=12) "www.nike.com",
 Form: (url.Values) <nil>,
 PostForm: (url.Values) <nil>,
 MultipartForm: (*multipart.Form)(<nil>),
 Trailer: (http.Header) <nil>,
 RemoteAddr: (string) "",
 RequestURI: (string) "",
 TLS: (*tls.ConnectionState)(<nil>),
 Cancel: (<-chan struct {}) <nil>
})
2016/06/22 16:56:57 ----------------------------
2016/06/22 16:56:57 StatusCode 403
(*http.Request)(0xc820110000)({
 Method: (string) (len=3) "GET",
 URL: (*url.URL)(0xc8200ea180)(http://www.nike.com/robots.txt),
 Proto: (string) (len=8) "HTTP/1.1",
 ProtoMajor: (int) 1,
 ProtoMinor: (int) 1,
 Header: (http.Header) (len=1) {
  (string) (len=10) "User-Agent": ([]string) (len=1 cap=1) {
   (string) (len=104) "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
  }
 },
 Body: (io.ReadCloser) <nil>,
 ContentLength: (int64) 0,
 TransferEncoding: ([]string) <nil>,
 Close: (bool) false,
 Host: (string) (len=12) "www.nike.com",
 Form: (url.Values) <nil>,
 PostForm: (url.Values) <nil>,
 MultipartForm: (*multipart.Form)(<nil>),
 Trailer: (http.Header) <nil>,
 RemoteAddr: (string) "",
 RequestURI: (string) "",
 TLS: (*tls.ConnectionState)(<nil>),
 Cancel: (<-chan struct {}) <nil>
})
2016/06/22 16:56:57 ----------------------------

EDIT: I've been playing around a bit and if I set ALL the other headers that Chrome sends then it works.

Answer 1

Your test code uses the same HTTP client object to send both requests, which presumably both go over the same connection.

Did you try sending the requests on different connections? Try setting your flags to true then false and see if it's always just the second one that fails.