Pinging @elastic/es-search-aggs

Another question that popped up is whether we should return "precise" lower bounds or just indicate the lower lower bound. If track_total_hits is set to 100 for instance, returning 123+ doesn't bring much more than returning 100+. The lower bound computed on each shard is not an approximation so any result above 100 can be misleading, a simple term query that can skip a lot of documents may return a small lower bound even if this term is present in all documents.

I don't like changing the meaning of total or make it a string. We already return "total" : -1 if track_total_hits is false. We can add an additional parameter at_least to inform the user how many hits we can have at least. We can also add this easily to 6.x. From my perspective this would only guarantee best effort. On the request end I am in favor of option 1.

I'm also not a fan of the options where the meaning of total is context-dependent. I'd rather have a new field. In both cases you need to conditionally check two things, but at least this way incorrect assumptions are quickly noticed (total: -1) and it doesn't change the meaning of total.

It's also a bit more graceful upgrade path. UIs might be showing a bad value (-1) but it doesn't break code outright and users can implement the check for at_least/min/whatever when they get a chance.

Also prefer an additional property, total can already be skewed in the case of partial results (if allow_partial_search_results is not true (default false)) which throws people. A new property on the response is more explicit.

On the request side can we create a new property e.g track_total_hits_upto=NUMBER or total_accuracy=NUMBER and deprecatetrack_total_hits=BOOL instead of reusing track_total_hits ?

We had a long discussion about this issue, here are some arguments that have been made:

• we already have a big breaking change coming: the removal of types, maybe we should try to not introduce another major breaking change at the same time
• it's too late for breaking changes in 7.0, anything that would require clients to change behavior by default would have to wait for 8.0 anyway
• we want to enable this optimization by default eventually, because users should opt in to get more info at the expense of more CPU usage
• option 3 is similar to 4 but is easier to parse and feels better
• in general option 1 didn't get much support
• option 2 raised concerns around the fact that one needs to check the value of another field (out of many) to know how to interpret the value of hits.total
• option 0 and 2 have the benefit of not being breaking for apps that don't check hit counts
• often apps check the hit count to know whether they need to display something, so we need to be careful with not breaking apps that work like that

Even though we couldn't make a decision, we narrowed down to two main paths:

A. Keep same format

• In 7.0 we make track_total_hits accept numbers in addition to boolean, which acts as a threshold for the number of hits to count. If hits.total is greater than or equal to this number then its value is a lower bound of the actual hit count. hits.total is still -1 if track_total_hits is set to false like in 6.x.
• In 7.0 we issue deprecation warnings when not setting track_total_hits
• In 8.0 we change the default of track_total_hits to false.

One modified version of this plan that has been discussed is to not return hits.total at all when track_total_hits is set to false in 8.0, to be consistent with eg. aggs which don't have any value when not requested, and not confuse users starting with Elasticsearch that something may be broken if hits.total is not a number.

This path has the downside of making the transition easier but the response is less explicit: you need to know the threshold that was set in the request in order to know how to interpret hits.total in the response.

B. Make hits.total an object

• In 7.0 we make track_total_hits accept numbers in addition to boolean, which acts as a threshold for the number of hits to count. If set to a number then hits.total is an object (it remains an integer by default for bw compat) that looks like { "hits": { "total": { "value": 1234, "relation": "gte" } } }
• In version 7.0 we issue deprecation warnings when track_total_hits is not set to a number to force users to opt in for the new response format.
• In version 8.0 we can change the default value to potentially 1000 like Lucene or 0 to disable the tracking of hit counts entirely

This path requires more changes on client side and makes it harder to perform generic processing of search responses due to the fact that the format of hits.total will depend on the request in 7.x. Yet it makes responses more self-contained.

Just want to confirm my understanding of the latest plan:

Kibana will not have to do anything for 7.0 and everything will continue to work the same except we may see deprecation warnings.

If that changes (e.g. the decision to switch the default in 7.0 changes), or is incorrect, please let me know!

Sorry I thought that I commented in this issue but for some reasons my comment is not there anymore.
So the current status after the internal discussion we had is that we'll disable track_total_hits by default in 7.0 and make hits.total and object with total (an integer that contains the number of hits) and relation, a string that defines how the total should be interpreted. When track_total_hits is false (default behavior) the total is filled with -1. We also discussed a way to get a response with the old format and decided that the client could send the major version in the request (as an header for instance), 7 would mean new response format and would default to "track_total_hits":false whereas 6 would behave like a 6x cluster (track_total_hits:true and hits.total returned as a simple integer). For Kibana this means that you'll have to explicitly set track_total_hits to true or use this new client versioning. The former can be tested now since the track_total_hits option is already available in 6x.