dtsps00544 2017-11-10 05:10
浏览 115
已采纳

如何将HTTP内容流式传输到文件和缓冲区?

I want to copy the data from an http response to a file and a buffer.

However, I can't quite figure this out.

Initially I had this:

func DownloadS3(hash string, cluster Cluster, tok *oauth.Token, offset, length int64, f *os.File) ([]byte, error) {

    // ... other code not shown ...

    resp, err = DoHTTPRequest("GET", s3tok.URL, nil, reqhdr, defaultClientTimeout)
    if err != nil {
        fmt.Println("Error downloading from cluster:", err)
        return nil, err
    }
    defer resp.Body.Close()

    if resp.StatusCode != 200 && resp.StatusCode != 206 {
        return nil, err
    }

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)

    // This actually does copy the whole downloaded data to the file as
    // expected.  However, I didn't expect io.CopyBuffer to not use the entire
    // buffer.  It ends up returning a portion of the file.
    _, err = io.CopyBuffer(f, resp.Body, buf)

    return buf, err
}

So what I want actually is something like

_, err = io.CopyBuffer(f, io.TeeReader(resp.Body, buf), nil)

Only, I cant pass buf into TeeReader as it doesn't implement the writer interface. I'm sure there is a proper method but I can't find it as I fumble around looking for an efficient way to do this.

How do I do this without allocating buffer after buffer. I was trying to be efficient. i.e. It seems silly to write the file and read it back.

What I've tried but doesn't work as expected.

    // Create buffer to return actual content
    buf := make([]byte, resp.ContentLength)
    _, err = io.CopyBuffer(f, io.TeeReader(resp.Body,bytes.NewBuffer(buf)), nil)
    return buf, nil
  • 写回答

2条回答 默认 最新

  • dqssst0144 2017-11-10 13:08
    关注

    Just to make the (other) answer clear and complete: it's not io.TeeReader()'s fault that you were not able to copy the complete body to file and have it as a bytes.Buffer, but it's entirely io.CopyBuffer()'s fault.

    io.Copy() is the knight and shining armor, the one who keeps on copying until the whole input is consumed:

    Copy copies from src to dst until either EOF is reached on src or an error occurs.

    So an equivalently good solution using io.TeeReader():

    buf := bytes.NewBuffer(make([]byte, 0, resp.ContentLength))
    _, err = io.Copy(f, io.TeeReader(resp.Body, buf))
    return buf.Bytes(), err
    

    Using bytes.NewBuffer() like this you can pre-allocate the necessary buffer and avoid reallocation and copying (which makes it faster). Of course you can use this with io.MultiWriter() too.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(1条)

报告相同问题?

悬赏问题

  • ¥20 请问这种量表怎么用spss量化分析(作为中介模型的因变量
  • ¥55 AD844 howland电流源如何驱动大额负载
  • ¥15 C++ /QT 内存权限的判断函数列举
  • ¥15 深度学习GFnet理解问题
  • ¥15 单细胞小提琴堆叠图代码
  • ¥50 升级strust2版本到2.3.15.1后使用ognl3.0.6.jar windows环境中没有问题,但部署到linux环境报错
  • ¥15 vue页面,node封装接口
  • ¥15 求TMS320F280039C工程模板!
  • ¥15 delphi+fastreport实现分组补空打印问题
  • ¥15 使用python把两台mysql数据库服务器数据导出和导入