dpvmtdu364462
2016-05-31 11:39
浏览 220
已采纳

在Go中,如何有效地将流式HTTP响应正文写入文件中的搜索位置?

I have a program that combines multiple http responses and writes to the respective seek positions on a file. I am currently doing this by

client := new(http.Client)
req, _ := http.NewRequest("GET", os.Args[1], nil)
resp, _ := client.Do(req)
defer resp.Close()
reader, _ := ioutil.ReadAll(resp.Body) //Reads the entire response to memory
//Some func that gets the seek value someval
fs.Seek(int64(someval), 0)
fs.Write(reader)

This sometimes results in a large memory usage because of the ioutil.ReadAll.

I tried bytes.Buffer as

buf := new(bytes.Buffer)
offset, _ := buf.ReadFrom(resp.Body) //Still reads the entire response to memory.
fs.Write(buf.Bytes())

but it was still the same.

My intention was to use a buffered write to the file, then seek to the offset again, and to continue write again till the end of stream is received (and hence capturing the offset value from buf.ReadFrom). But it was also keeping everything in the memory and writing at once.

What is the best way to write a similar stream directly to the disk, without keeping the entire content in buffer?

An example to understand would be much appreciated.

Thank you.

图片转代码服务由CSDN问答提供 功能建议

我有一个程序,该程序结合了多个http响应并写入文件中的相应查找位置。 我目前正在

  client进行此操作:= new(http.Client)
req,_:= http.NewRequest(“ GET”,os.Args [1],  nil)
resp,_:= client.Do(req)
defer resp.Close()
reader,_:= ioutil.ReadAll(resp.Body)//读取对内存的整个响应
 //某些函数 获取搜索值someval 
fs.Seek(int64(someval),0)
fs.Write(reader)
   
 
 

这有时会导致大量内存使用,因为 ioutil.ReadAll 的代码。

我尝试了 bytes.Buffer 作为

  buf:= new(bytes.Buffer)
offset,  _:= buf.ReadFrom(resp.Body)//仍然读取对内存的整个响应。
fs.Write(buf.Bytes())
   
 
 

但是 还是一样。

我的意图是使用对文件的缓冲写操作,然后再次查找偏移量,并再次继续写操作,直到接收到流的末尾为止(并因此从 buf.ReadFrom)。 但是它也将所有内容保存在内存中并立即写入。

将相似的流直接写入磁盘而不将整个内容保留在缓冲区中的最佳方法是什么?

一个可以理解的示例将是 非常感谢。

谢谢。

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • douzang7928 2016-05-31 12:06
    已采纳

    Use io.Copy to copy the response body to the file:

    resp, _ := client.Do(req)
    defer resp.Close()
    //Some func that gets the seek value someval
    fs.Seek(int64(someval), 0)
    n, err := io.Copy(fs, resp.Body)
    // n is number of bytes copied
    
    已采纳该答案
    打赏 评论

相关推荐 更多相似问题