从elasticsearch查询返回所有文档

Question

My question is specific to the "gopkg.in/olivere/elastic.v2" package I am using.

I am trying to return all documents that match my query:

termQuery := elastic.NewTermQuery("item_id", item_id)
searchResult, err := es.client.Search().
    Index(index).
    Type(SegmentsType). // search segments type
    Query(termQuery).   // specify the query
    Sort("time", true). // sort by "user" field, ascending
    From(0).Size(9).
    Pretty(true). // pretty print request and response JSON
    Do()          // execute
if err != nil {
    // Handle error
    return timeline, err
}

The problem is I get an internal server error if I increase the size to something large. If I eliminate the line that states:

From(0).Size(9).

then the default is used (10 documents). How may I return all documents?

Answer 1

Using a scroller to retrieve all results is just a bit different and in the interest of brevity I'm not including much error handling that you might need.

Basically you just need to slightly change your code from Search to Scroller and then loop with the Scroller calling Do and handling pages of results.

termQuery := elastic.NewTermQuery("item_id", item_id)
scroller := es.client.Scroller().
    Index(index).
    Type(SegmentsType). 
    Query(termQuery).   
    Sort("time", true). 
    Size(1)

docs := 0
for {
    res, err := scroller.Do(context.TODO())
    if err == io.EOF {
        // No remaining documents matching the search so break out of the 'forever' loop
        break
    }
    for _, hit := range res.Hits.Hits {
        // JSON parse or do whatever with each document retrieved from your index
        item := make(map[string]interface{})
        err := json.Unmarshal(*hit.Source, &item)
        docs++
    }
}