在ElasticSearch上使用特定搜索类型进行分页

Question

We are currently using ElasticSearch 6.7 and have a huge amount of data making some request taking too much time. To avoid this problem, we want to set up pagination within our research towards elasticsearch. The problem is that I can't put one of the pagination methods proposed by ES on the different requests that already exist. For example, this request contains different aggregations and a query: https://github.com/trackit/trackit/blob/master/usageReports/lambda/es_request_constructor.go#L61-L75 In addition, the results are sorted after the information is collected.

I tried to set up the Search After method as well as a form of pagination using from & size. Scroll doesn't works with aggregations and composite aggregation doesn't accept query.

So, there is any good way to do pagination in ElasticSearch combined with other request type and how to do it with the example above?

Answer 1

composite aggregation doesn't accept query

It does accept query. In the example below, the results are filtered based on play_name. The aggregation only get applied to the result of the query and it can be paginated using the after option.

{
    "query": {
        "term": {
            "play_name": "A Winters Tale"
        }
    },
    "size": 0,
    "aggs": {
        "speaker": {
            "composite": {
                "after": {
                    "product": "FLORIZEL"
                },
                "sources": [
                    {
                        "product": {
                            "terms": {
                                "field": "speaker"
                            }
                        }
                    }
                ]
            },
            "aggs": {
                "speech_number": {
                    "terms": {
                        "field": "speech_number"
                    },
                    "aggs": {
                        "line_id": {
                            "terms": {
                                "field": "line_id"
                            }
                        }
                    }
                }
            }
        }
    }
}