在Go中请求多个URL

Question

I have the following Go program: https://play.golang.org/p/-TUtJ7DIhi

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "strconv"
)

func main() {
    body, err := get("https://hacker-news.firebaseio.com/v0/topstories.json")

    if err != nil {
        panic(err)
    }

    var ids [500]int
    if err = json.Unmarshal(body, &ids); err != nil {
        panic(err)
    }

    var contents []byte
    for _, value := range ids[0:10] {
        body, err := get("https://hacker-news.firebaseio.com/v0/item/" + strconv.Itoa(value) + ".json")

        if err != nil {
            fmt.Println(err)
        } else {
            contents = append(contents, body...)
        }
    }

    fmt.Println(contents)
}

func get(url string) ([]byte, error) {
    res, err := http.Get(url)
    if err != nil {
        return nil, err
    }

    body, err := ioutil.ReadAll(res.Body)
    res.Body.Close()

    return body, err
}

When run it throws EOF json errors on the iterative get requests, but when I hit the URLs individually they do not appear to be malformed.

What am I missing?

Answer 1

It looks like there's something wrong with their server, and it's closing connections without sending a Connection: close header. The client therefore tries to reuse the connection per the HTTP/1.1 specification.

You can work around this by creating your own request, and setting Close = true, or using a custom Transport with DisableKeepAlives = true

req, err := http.NewRequest("GET", url, nil)
if err != nil {
    return nil, err
}
req.Close = true

res, err := http.DefaultClient.Do(req)
if err != nil {
    return nil, err
}