Golang：使用Go Routines和JSON进行HTTP调用并解析JSON

Question

I am relatively new to golang, and I wanted to create a way to do concurrently call multiple URLs, and parse the JSON documents. However, I'm really unsure if I am properly using go routines and channels. At this point, I'm not sure if I am not properly "thinking in Go" or if my understanding/approach of goroutines and channels is not accurate.

Additionally, when parsing, I would like to parse the results property from the body, which is an array, and each element in results contains a doc property that I'd like to filter out.

The goal is to do multiple fetches, concurrently and parse the responses for just the doc property inside of an responses body results array.

Definitely would appreciate any insight or suggestions to understand things better. Thanks in advance.

package operations

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "os"
    "strings"
)

// CouchbaseDoc parses .doc property from sync gateway documents
type CouchbaseDoc struct {
    Doc map[string]string `json:"doc"`
}

// Results deconstruct... results is a property of body, and is an array of obects
type Results struct {
    Results []byte `json:"results"`
}

func createURLs(channels []string) map[string]string {
    urlMap := make(map[string]string)

    domain := "swap" + strings.TrimSpace(os.Getenv("env"))
    bucket := strings.TrimSpace(os.Getenv("bucket"))
    for _, doctype := range channels {
        urlMap[doctype] = fmt.Sprintf("https://%s.endpoint.com/%s/_changes?filter=sync_gateway/bychannel&channels=%s&include_docs=true", domain, bucket, doctype)
    }

    return urlMap
}

func getChangesFeed(url string, ch chan map[string]string) {
    resp, _ := http.Get(url)

    body, _ := ioutil.ReadAll(resp.Body)

    go parseBody(body, ch)
}

func parseBody(body []byte, ch chan map[string]string) {
    var results Results
    var doc CouchbaseDoc
    json.Unmarshal(body, &results)
    json.Unmarshal(results.Results, &doc)
    // write to responses
    ch <- doc.Doc
}

func fetchDocs(channels []string) {
    urls := createURLs(channels)

    // Response channel is where all go routines will do the dirty
    responses := make(chan map[string]string)
    for _, url := range urls {
        go getChangesFeed(url, responses)
    }

    // Read from responses channel
    docs := <-responses
    for doc := range docs {
        fmt.Println(doc) // This should print results ??
    }
}

Answer 1

The Fix

This line:

docs := <-responses

Will only receive one element from the channel, not all elements. However, you could call the receive operation once per expected send on the channel, this would be the simplest fix for your code:

responses := make(chan map[string]string)
for _, url := range urls {
    go getChangesFeed(url, responses)
}

for x := 0; x < len(urls); x++ {
    fmt.Println(<-responses)
}

More Information

Note that you are using an unbuffered channel because you did not give the channel a length. A for e := range ch { loop is only appropriate for a buffered channel, and only after the buffered channel has been closed.

Closing a buffered channel indicates no more data will be sent on the channel, and probably wouldn't match your program design well (especially without a sync.WaitGroup).

So using the buffered channel is fine: you just have to know that neither the send nor the receive operation will proceed unless both sides are ready: meaning each is blocked and waiting for the other.

This was easily accomplished with the above code by putting the sends in goroutines and queueing up an equal number of receive operations in the main goroutine using a loop with an equal counter.

To learn more, read The Language Specification and Effective Go and the Mutex and WaitGroup sections of the sync package documentation.

Runnable Demonstration

Here's a complete, runnable example that demonstrates the principle:

package main

import(
    "fmt"
    "time"
)

func Sleep1(ch chan int) {
    time.Sleep(time.Second)
    ch <- 1
}

func Sleep3(ch chan int) {
    time.Sleep(time.Second * 3)
    ch <- 3
}

func Sleep5(ch chan int) {
    time.Sleep(time.Second * 5)
    ch <- 5
}

func main() {
    ch := make(chan int)
    go Sleep1(ch)
    go Sleep3(ch)
    go Sleep5(ch)
    for x := 0; x < 3; x++ {
        fmt.Println(<-ch)
    }
}