dougong7850
2018-10-10 18:53
浏览 334
已采纳

我如何一次从包含json数组的流中读取一个json对象,而不将整个数组加载到内存中?

To reduce memory usage and reduce latency i want to start processing the json objects returned in a http response before they have all been sent. I am trying to write a method on my web service client that will query a web service returning a channel that will feed the structs from the web request as the come off the stream.

    func (c *Client) GetMyObj() (<-chan models.MyObj, <-chan error) {
       outChan := make(chan models.MyObj)
       errChan := make(chan error)
       go func() {
          resp, err := httpClient.Get(c.ServiceUrl, "text/plain;charset=UTF-8", nil)
          if err != nil {
            errChan <- err
            return
          }
          if resp.Status != "200 OK" {
             errChan <- fmt.Errorf("http error: %1", resp.Status)
             return
          }

          dec := json.NewDecoder(resp.Body)

          for dec.More() {
             myO:= models.MyObj{}
             err := dec.Decode(&MyO)
             if err != nil {
                errChan <- err
                return
             }
             outChan <- dfe
          }           
       }()

       return outChan, errChan
    }

Unfortunately this produces an exception when it reads the preceding opening square bracket.The json document would look something like the below but with many more objects in the array and more properties on the object, but you get the idea:

    [{"name":"Value"}
    ,{"name":"Value"}
    ,{"name":"Value"}]
  • 写回答
  • 好问题 提建议
  • 追加酬金
  • 关注问题
  • 邀请回答

1条回答 默认 最新

  • dougao1106 2018-10-10 20:21
    最佳回答

    The example listed in thejson.Decoder.Decode(...) method documentation shows exactly this.

    In short, you do the following steps:

    1. Read a token using dec.Token() (and optionally expect it to be an open square bracket)
    2. Read tokens using dec.Decode() while dec.More()
    3. Optionally, read the final closing square bracket.

    For example (Go Playground):

    dec := json.NewDecoder(jsonStream)
    type Item struct{ Name string }
    
    // Read the open bracket.
    t, err := dec.Token()
    if err != nil {
        panic(err)
    }
    fmt.Printf("OK: %T: %v
    ", t, t)
    
    // While the array contains values.
    for dec.More() {
        // Decode an array value.
        var item Item
        err := dec.Decode(&item)
        if err != nil {
            panic(err)
        }
        fmt.Printf("OK: item=%#v
    ", item)
    }
    
    // Read the closing bracket.
    t, err = dec.Token()
    if err != nil {
        panic(err)
    }
    fmt.Printf("OK: %T: %v
    ", t, t)
    // OK: json.Delim: [
    // OK: item=main.Item{Name:"Value1"}
    // OK: item=main.Item{Name:"Value2"}
    // OK: item=main.Item{Name:"Value3"}
    // OK: json.Delim: ]
    
    评论
    解决 无用
    打赏 举报