douwen7905
douwen7905
2016-08-10 13:20

限制对HTTP GET请求的响应中读取的数据量

已采纳

I'm scraping HTML pages and have set up a HTTP client like so:

client := *http.Client{
        Transport: &http.Transport{
            Dial: (&net.Dialer{
                Timeout:   30 * time.Second,
                KeepAlive: 30 * time.Second,
            }).Dial,
            TLSHandshakeTimeout:   10 * time.Second,
            ResponseHeaderTimeout: 10 * time.Second,
        },
    }

Now when I make GET requests of multiple URLs I don't want to get stuck with URLs that deliver massive amount of data.

response, err := client.Get(page.Url)
checkErr(err)
body, err := ioutil.ReadAll(response.Body)
checkErr(err)
page.Body = string(body)

Is there a way to limit the amount of data (bytes) the GET request accepts from a resource and stops?

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

1条回答

  • drfals1307 drfals1307 5年前

    Use an io.LimitedReader

    A LimitedReader reads from R but limits the amount of data returned to just N bytes.

    limitedReader := &io.LimitedReader{R: response.Body, N: limit}
    body, err := ioutil.ReadAll(limitedReader)
    

    or

    body, err := ioutil.ReadAll(io.LimitReader(response.Body, limit))    
    
    点赞 评论 复制链接分享

相关推荐