如何将HTTP内容流式传输到文件和缓冲区？

Question

I want to copy the data from an http response to a file and a buffer.

However, I can't quite figure this out.

Initially I had this:

func DownloadS3(hash string, cluster Cluster, tok *oauth.Token, offset, length int64, f *os.File) ([]byte, error) {

    // ... other code not shown ...

    resp, err = DoHTTPRequest("GET", s3tok.URL, nil, reqhdr, defaultClientTimeout)
    if err != nil {
        fmt.Println("Error downloading from cluster:", err)
        return nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200 && resp.StatusCode != 206 {
        return nil, err
    }

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)

    // This actually does copy the whole downloaded data to the file as
    // expected.  However, I didn't expect io.CopyBuffer to not use the entire
    // buffer.  It ends up returning a portion of the file.
    _, err = io.CopyBuffer(f, resp.Body, buf)

    return buf, err
}

So what I want actually is something like

_, err = io.CopyBuffer(f, io.TeeReader(resp.Body, buf), nil)

Only, I cant pass buf into TeeReader as it doesn't implement the writer interface. I'm sure there is a proper method but I can't find it as I fumble around looking for an efficient way to do this.

How do I do this without allocating buffer after buffer. I was trying to be efficient. i.e. It seems silly to write the file and read it back.

What I've tried but doesn't work as expected.

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)
    _, err = io.CopyBuffer(f, io.TeeReader(resp.Body,bytes.NewBuffer(buf)), nil)
    return buf, nil

Answer 1

Just to make the (other) answer clear and complete: it's not io.TeeReader()'s fault that you were not able to copy the complete body to file and have it as a bytes.Buffer, but it's entirely io.CopyBuffer()'s fault.

io.Copy() is the knight and shining armor, the one who keeps on copying until the whole input is consumed:

Copy copies from src to dst until either EOF is reached on src or an error occurs.

So an equivalently good solution using io.TeeReader():

buf := bytes.NewBuffer(make([]byte, 0, resp.ContentLength))
_, err = io.Copy(f, io.TeeReader(resp.Body, buf))
return buf.Bytes(), err

Using bytes.NewBuffer() like this you can pre-allocate the necessary buffer and avoid reallocation and copying (which makes it faster). Of course you can use this with io.MultiWriter() too.