如何通过代理进行HTTP / HTTPS GET

Question

I'm trying to issue an http/s request via a working proxy. I have a working example in a legacy project in node JS, where using the native Node.js https (require('https')) lib can make the request if used with the following options object:

{
    host: "<actual target url>"
    hostname: "<proxy ip>"
}

For example, to make an https request to example.com via proxy 1.1.1.1, I'll use:

{
    host: "http://example.com"
    hostname: "1.1.1.1"
}

---

In Golang, I've tried several of the documented options. Specifically, I'd expect:

proxyUrl, _ := url.Parse("<proxy ip>")
myClient := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxyUrl)}}
resp, err := myClient.Get("https://<actual target url>/...")

The result is the proxy rejecting the connection, indicating something incorrect in the request. (err exists, and resp is nil)

---

The proxy itself is an NGINX instance configured according to this gist:

https://gist.github.com/selfish/6e858eb17aa82971d25b21775e9649cb#file-nginx-conf

Can anyone help understand the difference in HTTP handling for Node.js and Golang?

Answer 1

You are confusing forward and reverse proxies.

Conceptually it works like this:

Reverse proxies

• are deployed near the server side of a connection
• pretend to be the origin
• are controlled by or on behalf of the site owner
• are not inherently known to the user agent

Forward proxies

• are deployed near the client side of a connection
• are controlled by or on behalf of the user agent
• are explicitely configured in the user agent

(Reality is more complicated than that, of course, but this is sufficient to highlight the differences).

                                     Internet                    ||  Invisible to
                                         +                       ||  User Agent
                                         |                       ||
 +------------+     +---------------+    |    +---------------+  ||  +--------+
 |            |     |               |    |    |               |  ||  |        |
 | User Agent +---->+ Forward Proxy +-------->+ Reverse Proxy +----->+ Origin |
 |            |     |               |    |    |               |  ||  |        |
 +------------+     +---------------+    |    +---------------+  ||  +--------+
                                         |                       ||
                                         +                       ||
                                                                 ||

nginx is a reverse proxy, but by setting the Transport.Proxy field you treat it like a forward proxy. This is the request that nginx sees:

CONNECT example.com:443 HTTP/1.1
Host: example.com:443
User-Agent: Go-http-client/1.1

This essentialy means, "Establish a TCP connection to example.com:443 and then act like a dumb TCP proxy." Since nginx is a reverse proxy only it is rightfully confused when confronted with a CONNECT request.

To send a request to a particular reverse proxy you simply have to modify the request URL, and possibly the Host header (that depends on whether or not nginx expects a particular server_name). No special client configuration is required.

Assuming nginx runs on 198.51.100.1:

req, _ := http.NewRequest("GET", "http://198.51.100.1", nil)
req.Host = "example.com" // if necessary
res, _ := http.DefaultClient.Do(req)

This causes the following request to be sent to 198.51.100.1:80:

GET / HTTP/1.1
Host: example.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

Note that it is entirely up to the reverse proxy if the request actually hits example.com. The client has no knowledge or control over what happens after the proxy.

If you are not in a position to change the request, you can set the Transport.DialContext function such that your proxy is always dialed, independent of the request URL and Host header. This results in the same request as above and should be equivalent to your JavaScript code:

c := &http.Client{
    Transport: &http.Transport{
        DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
            return (&net.Dialer{}).DialContext(ctx, "tcp", "198.51.100.1:80")

            // Or 198.51.100.1:443 if nginx has TLS enabled, although that almost
            // certainly causes TLS validation errors because a certificate for 
            // example.com is expected.
        },
    },
}

req, _ := http.NewRequest("GET", "http://example.com", nil)
res, _ := c.Do(req)