在GO中以异步方式获取网址列表

Question

I my GO application, I often need to query a list or urls. As GO is async from ground up, it is the perfect place to use this functionality.

What is the best way solve that? I found a blog proposing a solution but it fails with an empty list of urls.

Thanks!

Answer 1

I've tweaked the code in the blog link you provided and made it slightly more resilient to error.

The code below should compile, and should handle boundary cases such as an empty input urls slice.

package main

import (
    "fmt"
    "net/http"
    "os"
    "time"
)

const timeout time.Duration = 3 * time.Second

var urls = []string{
    "http://golang.org/",
    "http://stackoverflow.com/",
    "http://i.wanta.pony/", // Should error
}

type httpResponse struct {
    url      string
    response *http.Response
    err      error
}

func asyncHTTPGets(urls []string, ch chan *httpResponse) {
    for _, url := range urls {
        go func(url string) {
            resp, err := http.Get(url)
            ch <- &httpResponse{url, resp, err}
        }(url)
    }
}

func main() {
    responseCount := 0
    ch := make(chan *httpResponse)
    go asyncHTTPGets(urls, ch)
    for responseCount != len(urls) {
        select {
        case r := <-ch:
            if r.err != nil {
                fmt.Printf("Error %s fetching %s
", r.err, r.url)
            } else {
                fmt.Printf("%s was fetched
", r.url)
            }
            responseCount++
        case <-time.After(timeout):
            os.Exit(1)
        }
    }
}

Playground