扩展Golang的http.Resp.Body以处理大文件

Question

I have a client application which reads in the full body of a http response into a buffer and performs some processing on it:

body, _ = ioutil.ReadAll(containerObject.Resp.Body)

The problem is that this application runs on an embedded device, so responses that are too large fill up the device RAM, causing Ubuntu to kill the process.

To avoid this, I check the content-length header and bypass processing if the document is too large. However, some servers (I'm looking at you, Microsoft) send very large html responses without setting content-length and crash the device.

The only way I can see of getting around this is to read the response body up to a certain length. If it reaches this limit, then a new reader could be created which first streams the in-memory buffer, then continues reading from the original Resp.Body. Ideally, I would assign this new reader to the containerObject.Resp.Body so that callers would not know the difference.

I'm new to GoLang and am not sure how to go about coding this. Any suggestions or alternative solutions would be greatly appreciated.

Edit 1: The caller expects a Resp.Body object, so the solution needs to be compatible with that interface.

Edit 2: I cannot parse small chunks of the document. Either the entire document is processed or it is passed unchanged to the caller, without loading it into memory.

Answer 1

If you need to read part of the response body, then reconstruct it in place for other callers, you can use a combination of an io.MultiReader and ioutil.NopCloser

resp, err := http.Get("http://google.com")
if err != nil {
    return err
}
defer resp.Body.Close()

part, err := ioutil.ReadAll(io.LimitReader(resp.Body, maxReadSize))
if err != nil {
    return err
}

// do something with part

// recombine the buffered part of the body with the rest of the stream
resp.Body = ioutil.NopCloser(io.MultiReader(bytes.NewReader(part), resp.Body))

// do something with the full Response.Body as an io.Reader

If you can't defer resp.Body.Close() because you intend to return the response before it's read in its entirety, you will need to augment the replacement body so that the Close() method applies to the original body. Rather than using the ioutil.NopCloser as the io.ReadCloser, create your own that refers to the correct method calls.

type readCloser struct {
    io.Closer
    io.Reader
}

resp.Body = readCloser{
    Closer: resp.Body,
    Reader: io.MultiReader(bytes.NewReader(part), resp.Body),
}