dongyan4424
2015-08-05 12:18
浏览 66
已采纳

大量内容的go io.copy函数中的一个错误始终一致

This one works consistently.

    _, err = io.Copy(out, resp.Body)
    if err != nil {
        ErrLog.Fatal(err)
    }

This one gives pretty consistent off by one errors (the last byte of the downloaded content is left off, in my case a closing ] in a json response) for large-ish responses (MBs).

    if _, err := io.Copy(out, resp.Body); err != nil {
        ErrLog.Fatal(err)
    }

From the examples on the official golang blog, it looks like this should be valid syntax.

Edit: Some more details and context

This is the error I get with the 2nd version of the code (more compact error handling)

ERROR: 2015/08/05 08:09:31 pull.go:257: unexpected end of JSON input

From this code in another function

err = json.Unmarshal(dat, &all_data)
if err != nil {
    return err
}

I found the off by one issue by looking at the first 10 and last 10 characters of the file in each case. Here are the before and afters:

# Before (with error)
START:  [{"tags":[              END:    ersion":1}
START:  [{"_create              END:    "tags":[]}

# After
START:  [{"tags":[              END:    rsion":1}]
START:  [{"_create              END:    tags":[]}]

The files are 15-20 Mb json strings.

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • duanlachu7344 2015-08-05 21:00
    已采纳

    It turned out the issue was at least partially the result of a race condition.

    I was not calling .Close() on the file out before exiting the function. After adding that I have not been any more issues.

    Why exactly this was causing just the last byte of the file to get dropped sometimes is a mystery to me.

    点赞 打赏 评论

相关推荐 更多相似问题