dr2898
2017-09-14 16:39
浏览 333
已采纳

扩展Golang的http.Resp.Body以处理大文件

I have a client application which reads in the full body of a http response into a buffer and performs some processing on it:

body, _ = ioutil.ReadAll(containerObject.Resp.Body)

The problem is that this application runs on an embedded device, so responses that are too large fill up the device RAM, causing Ubuntu to kill the process.

To avoid this, I check the content-length header and bypass processing if the document is too large. However, some servers (I'm looking at you, Microsoft) send very large html responses without setting content-length and crash the device.

The only way I can see of getting around this is to read the response body up to a certain length. If it reaches this limit, then a new reader could be created which first streams the in-memory buffer, then continues reading from the original Resp.Body. Ideally, I would assign this new reader to the containerObject.Resp.Body so that callers would not know the difference.

I'm new to GoLang and am not sure how to go about coding this. Any suggestions or alternative solutions would be greatly appreciated.

Edit 1: The caller expects a Resp.Body object, so the solution needs to be compatible with that interface.

Edit 2: I cannot parse small chunks of the document. Either the entire document is processed or it is passed unchanged to the caller, without loading it into memory.

图片转代码服务由CSDN问答提供 功能建议

我有一个客户端应用程序,它将HTTP响应的整个内容读入缓冲区并对其进行一些处理 :

body,_ = ioutil.ReadAll(containerObject.Resp.Body)

问题是此应用程序在嵌入式设备上运行,因此 太大的响应会填满设备RAM,导致Ubuntu终止进程。

为避免这种情况,如果文档太大,我会检查content-length标头并绕过处理。 但是,某些服务器(我正在向您询问,Microsoft)会发送非常大的html响应,而没有设置内容长度并使设备崩溃。

我唯一能解决这个问题的方法 是读取响应正文达一定长度。 如果达到此限制,则可以创建一个新的读取器,该读取器首先对内存缓冲区进行流处理,然后继续从原始Resp.Body中读取数据。 理想情况下,我会将这个新的阅读器分配给containerObject.Resp.Body,以便调用者不知道它们之间的区别。

我是GoLang的新手,不确定如何进行编码 这个。 任何建议或替代解决方案将不胜感激。

编辑1:调用者需要一个Resp.Body对象,因此解决方案必须与该接口兼容。

编辑2:我无法解析文档的小块。 要么处理整个文档,要么将其原封不动地传递给调用者,而不将其加载到内存中。

  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • douboshan1466 2017-09-14 17:28
    已采纳

    If you need to read part of the response body, then reconstruct it in place for other callers, you can use a combination of an io.MultiReader and ioutil.NopCloser

    resp, err := http.Get("http://google.com")
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    
    part, err := ioutil.ReadAll(io.LimitReader(resp.Body, maxReadSize))
    if err != nil {
        return err
    }
    
    // do something with part
    
    // recombine the buffered part of the body with the rest of the stream
    resp.Body = ioutil.NopCloser(io.MultiReader(bytes.NewReader(part), resp.Body))
    
    // do something with the full Response.Body as an io.Reader
    

    If you can't defer resp.Body.Close() because you intend to return the response before it's read in its entirety, you will need to augment the replacement body so that the Close() method applies to the original body. Rather than using the ioutil.NopCloser as the io.ReadCloser, create your own that refers to the correct method calls.

    type readCloser struct {
        io.Closer
        io.Reader
    }
    
    resp.Body = readCloser{
        Closer: resp.Body,
        Reader: io.MultiReader(bytes.NewReader(part), resp.Body),
    }
    
    已采纳该答案
    打赏 评论

相关推荐 更多相似问题