io.copyN在第一次不被调用时无法工作

Question

I'm trying to download a image from a site, and the steps are as follow:

• use http.Get to fetch the image
• use os.Create to create a new file in current folder
• use io.copyN to copy the image into the file

But it is weird if the io.CopyN is failed at the first time, it seems never success later

code fragment：

    download_again:
        copy_byte, copy_err := io.CopyN(file, res.Body, res.ContentLength)
        fmt.Fprintf(os.Stderr, "img(%s) size: %d
", name, res.ContentLength)
        if copy_err == nil && res.ContentLength == copy_byte {
            fmt.Fprintf(os.Stderr, "Success to download img(%s)[%f KB(%d B)]: %s
", img_url, float64(copy_byte)/1024, copy_byte, name)
        } else {
            if try_i > download_times {
                fmt.Fprintf(os.Stderr, "[fatal] fail to download img(%s) %s
", img_url, name)
                fout.WriteString(name + "
")
                return
            }
            fmt.Fprintf(os.Stderr, "error in download img(%s)[%f KB(%d B)]: %s, try %d times
", img_url, float64(copy_byte)/1024, copy_byte, name, try_i)
            try_i++
            goto download_again
        }

and the output message:

img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[171.447266 KB(175562 B)]: 11085 , try 1 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085 , try 2 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 3 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 4 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 5 times
img(11085) size: 273047
error in download img(./style/images/dszp/11085.jpg)[0.000000 KB(0 B)]: 11085, try 6 times

Any help will be appreciated :)

Answer 1

You need to retry the http request. (not just the io.CopyN)

Here is a (mostly untested) implementation of a resumable download:

func downloadFile(dst *os.File, url string, offset int64) (int64, error) {
    req, err := http.NewRequest("GET", url, nil)
    if err != nil {
        return 0, err
    }
    // try to use a range header
    if offset > 0 {
        req.Header.Set("Range", fmt.Sprintf("bytes=%d-", offset))
    }
    res, err := http.DefaultClient.Do(req)
    if err != nil {
        return 0, err
    }
    defer res.Body.Close()

    // some http servers don't support range, so in that case we discard the first
    // offset bytes
    if offset > 0 && res.StatusCode != http.StatusPartialContent {
        _, err = io.CopyN(ioutil.Discard, res.Body, offset)
        if err != nil {
            return offset, err
        }
    }

    n, err := io.CopyN(dst, res.Body, res.ContentLength)
    return offset + n, err
}

You can pass that function an offset and it will try to pick up from that point (either by telling the HTTP server, or by discarding the bytes). It also returns the progress you've made, so you can then wrap this function in a retry loop:

func example() error {
    f, err := os.Create("/tmp/pkg.png")
    if err != nil {
        return err
    }
    defer f.Close()

    offset := int64(0)
    delay := time.Second
    // try 5 times
    for i := 0; i < 5; i++ {
        offset, err = downloadFile(f, "http://golang.org/doc/gopher/pkg.png", offset)
        if err == nil {
            return nil
        }
        // wait a little while before trying again
        time.Sleep(delay)
        delay *= 2
    }
    return fmt.Errorf("failed to download file after 5 tries")
}

Incidentally, you should really avoid using goto for loops.