为什么`nbytes，err：= io.Copy（ioutil.Discard，resp.Body）`总是返回0？

Question

I've written a simple program that fetch a list of url to store them inside some files. In this example google and gmail. I always run same command in different software version. Program is stored inside goFetchAll: this is the name of compiled version of the algorithm.

• go build
• ./goFetchAll http://www.google.com http://www.gmail.com

0.23s        0  http://www.google.com
1.15s        0  http://www.gmail.com

The second number should be the number of bytes of content. But it is alway 0.

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "os"
    "strings"
    "time"
)

func main() {
    start := time.Now()

    ch := make(chan string)

    for _, url := range os.Args[1:] {
        go fetch(url, ch)
    }

    for range os.Args[1:] {
        fmt.Println(<-ch)
    }

    secs := time.Since(start).Seconds()
    fmt.Sprintf("%.2fs elapsed
", secs)
}

func fetch(url string, ch chan<- string) {
    start := time.Now()
    resp, err := http.Get(url)
    if err != nil {
        ch <- fmt.Sprint(err)
        return
    }

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        ch <- fmt.Sprintf("Cant catch content")
        return
    }

    nbytes, err := io.Copy(ioutil.Discard, resp.Body)
    defer resp.Body.Close()
    if err != nil {
        ch <- fmt.Sprintf("while reading %s: %v", url, err)
        return
    }

    secs := time.Since(start).Seconds()
    ch <- fmt.Sprintf("%.2fs  %7d  %s", secs, nbytes, url)

    // store on file
    filename := string(url)
    filename = strings.Replace(filename, ":", "", -1)
    filename = strings.Replace(filename, "//", "-", -1)
    filename = strings.Replace(filename, "/", "", -1)
    filename = strings.Replace(filename, ".", "-", -1)
    filename = "downloads/" + filename + ".html"

    f, err := os.Create(filename)
    f.Write(body)
    defer f.Close()
    if err != nil {
        ch <- fmt.Sprintf("while writing %s: %v", url, err)
        return
    }
}

I've also an older version of same script that actually works:

0.25s    10363  http://www.google.com
0.89s    66576  http://www.gmail.com

package main

import (
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "os"
    "time"
)

func main() {
    start := time.Now()

    ch := make(chan string)

    for _, url := range os.Args[1:] {
        go fetch(url, ch)
    }

    for range os.Args[1:] {
        fmt.Println(<-ch)
    }

    fmt.Println("%.2fs elapsed
", time.Since(start).Seconds())
}

func fetch(url string, ch chan<- string) {
    start := time.Now()
    resp, err := http.Get(url)
    if err != nil {
        ch <- fmt.Sprint(err)
        return
    }

    nbytes, err := io.Copy(ioutil.Discard, resp.Body)
    resp.Body.Close()
    if err != nil {
        ch <- fmt.Sprintf("whioe reading %s: %v", url, err)
        return
    }

    secs := time.Since(start).Seconds()
    ch <- fmt.Sprintf("%.2fs  %7d  %s", secs, nbytes, url)
}

Can someone explain why the newest version always count 0 seconds?

---

My partial solution is the following. I've just request again http.Get(url)

resp, err := http.Get(url)
nbytes, err := io.Copy(ioutil.Discard, resp.Body)
defer resp.Body.Close() // dont leak resources
if err != nil {
    ch <- fmt.Sprintf("while reading %s: %v", url, err)
    return
}

resp, err = http.Get(url)
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
    ch <- fmt.Sprintf("Cant catch content")
    return
}

Answer 1

The reason for this is because you've already read the response at the time of that call once. So the second time, 0 bytes are read from the stream. After the error checking calls are removed:

resp, err := http.Get(url)
body, err := ioutil.ReadAll(resp.Body)
nbytes, err := io.Copy(ioutil.Discard, resp.Body)

Note the ReadAll call on the second line.

One more small suggestion I'd like to propose (not related to the actual question) is to use the defer calls right after initialising the stream. For instance:

resp, err := http.Get(url)
if err != nil {
    ch <- fmt.Sprint(err)
    return
}
defer resp.Body.Close()

Although not mentioned specifically, it can be inferred from this section in Effective Go. Paraphrasing here:

Second, it means that the close sits near the open, which is much clearer than placing it at the end of the function.