在每个请求上使用不同的代理

Question

As far as it is my understanding, the only way I can set a proxy for a request is to assign it to a client and then use that for a request. However, I also understand I should reuse the clients for my requests. Is there any way to individually assign proxies for each request whilst using the same client? Or any way to efficiently do a collection of requests with a different proxy for each request?

Answer 1

Creating of http.Client is not so expensive - its just a struct allocation that can be done for each request.

You should create http.Client and transport objects yourself and set proxy directly there.

proxyUrl, err := url.Parse("http://127.0.0.1:123")
if err != nil {
    panic(err)
}
myClient := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxyUrl)}}
myClient.Get("http://www.google.com/")

Tere is the other posibility to use always the same client with different proxies:

http.ProxyURL call from the first example is function:

// ProxyURL returns a proxy function (for use in a Transport)
// that always returns the same URL.
func ProxyURL(fixedURL *url.URL) func(*Request) (*url.URL, error) {
    return func(*Request) (*url.URL, error) {
        return fixedURL, nil
    }
}

You can extend it and chose proxy based on the information from request or implement some other logic.

UPD: here is implementation how to use ProxyUrl with only one Client. But overehead of changing context is comparable with overhead on creating a client. I would use the first scenario.

// We must write our own Get (or some other) request and add somewere the info that this is a proxy request. You can do it via context of request.

func GetWithProxy(c *http.Client,proxyUrl *url.URL, url string) (resp *http.Response, err error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return nil, err
    }

    if proxyUrl != nil {
        req = req.WithContext(context.WithValue(context.Background(),"proxy",proxyUrl))
    }

    return c.Do(req)
}

// Organize client that can use this information:

proxyUrl, err := url.Parse("http://127.0.0.1:123")
if err != nil {
    panic(err)
}

myClient := &http.Client{Transport: &http.Transport{Proxy: func(req *http.Request) (*url.URL, error) {
    currentProxy,_ := req.Context().Value("proxy").(*url.URL)
    return currentProxy, nil
}}}

Use it:

GetWithProxy(myClient,nil,"http://www.google.com/")
GetWithProxy(myClient,proxyUrl,"http://www.google.com/")