I'm creating tools for my company to load test our system. I currently have tools written in Python but I'm exploring options using Go, hoping for efficiency and performance gains as we need to generate millions of users simultaneously (many thousands per box with many boxes) and every little bit counts. My users primarily make http calls against our system and I need a very lightweight and efficient http client.
For our Python tools, a previous employee had some functions manually dealing with and reusing Sockets and totally ignoring cookies. While I'm totally new to Go, I've compared multiple Go networking implementations so far and I've been happy with CPU and network usage so I don't think I need to go so far as to manually manage sockets yet, but RAM usage has been multiple times higher than in our Python solution. I have a suspicion that's due to these Go implementations automatically maintaining and managing cookies as other higher level Python libraries did the same thing. I'm looking for ways to have my http client disable/block/ignore all cookies to either solve my problem or rule out cookies as my memory bloat culprit.
I've used a simple net/http Get() call, fasthttp to create a client that reuses varying numbers of connections per hosts (1000+), and creating a net/http.Client with and without a custom Transport. I've used pprof to memory profile each. Without overriding cookies, my http.Client flow seems to have the least memory bloat and fasthttp has the most. I've tried to go through all their docs and searched SO and the web and can't find anything that explicitly ignores or blocks cookies. Closest I found was fasthttp's DelAllCookies() for both requests and responses, but calling those had no discernible effect on memory usage. I've looked into net/http/cookiejar to set a custom jar and policy for net/http.Client but that looks more like a way to store and use cookie data, which I don't want to do, and I don't see a way to set a policy to ignore cookies altogether.
// fasthttp implementation
var httpClient *fasthttp.Client
func init() {
httpClient = &fasthttp.Client{
MaxConnsPerHost: 1000,
}
}
func fastHTTPClient(method string, uri string) (string, time.Duration) {
req := fasthttp.AcquireRequest()
req.Header.SetMethod(method)
req.SetRequestURI(uri)
resp := fasthttp.AcquireResponse()
startTime := time.Now()
err := httpClient.DoTimeout(req, resp, targetDuration)
elapsed := time.Since(startTime)
if err != nil {
log.Panicln(err)
body := string(resp.Body())
req.Header.DelAllCookies()
resp.Header.DelAllCookies()
fasthttp.ReleaseRequest(req)
fasthttp.ReleaseResponse(resp)
return body, elapsed
}
body := string(resp.Body())
req.Header.DelAllCookies()
resp.Header.DelAllCookies()
fasthttp.ReleaseRequest(req)
fasthttp.ReleaseResponse(resp)
return body, elapsed
}
// net/http implementation
var tr = &http.Transport{}
var client = &http.Client{Transport: tr}
func netGet(method string, uri string) (string, time.Duration) {
startTime := time.Now()
resp, err := client.Get(uri)
elapsed := time.Since(startTime)
if err != nil {
log.Panic(err)
body, err2 := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err2 != nil {
log.Panic(err)
}
return string(body), elapsed
}
body, err := ioutil.ReadAll(resp.Body)
resp.Body.Close()
if err != nil {
log.Panicln(err)
}
return string(body), elapsed
}
Again, these work functionally but when I run thousands of these concurrently using boomer I end up using multiple GB of RAM vs 1-1.5 GB with our Python Sockets implementation. How can I block/ignore cookies or otherwise be sure cookies aren't what's eating up all my memory?
