Pinging @elastic/es-core-infra

We haven't exposed updateable doc values in Elasticsearch because they provide a trade-off that is hard to reason about. For instance say you update a single value of a single document. The next refresh will need to rewrite doc values for the entire segment that contains this document. If it would get exposed, there are chances that such updateable fields would be used for things like view counters, and I wouldn't be surprised that for some users doc values for all segments would need to be rewritten on every refresh, which would certainly cause write performance / scalability issues. I'm not saying we shouldn't do it at all, but it would at least require careful documentation that it isn't like an in-place update. There are other options that could be considered as well, like storing this data in some side-car data-structure so that it doesn't necessarily have to live by the rules of the Lucene index like the need to provide point-in-time snapshots.

Thanks @jpountz for explaining so clearly why we don't expose updateable doc values! I imagine a side car that doesn't follow the Lucene visibility rules might actually work for this but might be pretty confusing as well. I'd love for us to have something here because it is an important feature for folks that are concerned with optimizing search relevance based on frequently changing signals. Which feels like a thing we should support.

Yeah I'm unhappy that some users resort to using things like parent/child to solve this problem, by storing the frequently-changing values in small documents that they later join at search time. It introduces other issues. I wish we provided something better. Let me try to summarize the options that I am aware of:

Use the _update API

• simple, but
• need to reindex the entire document
• higer merging activity and (slightly) slower searches because of deletes

Use updateable doc values

• significant write amplification since doc values of the updated field need to be rewritten for entire segments
• fast at search time

Make doc values support stacked updates ie. writes would only write a delta and things would be resolved on read

• better write scalability
• slower search

Side-car data

• values could be updated in-place
• wouldn't support point-in-time snapshots

X-Pack Machine Learning currently does something very similar to the workaround in the original issue description when renormalizing anomaly scores. After renormalization we bulk index the result documents where the anomaly score has changed significantly, but leave existing results untouched where the change to the anomaly score is small.

So any changes that are made as a result of this issue could benefit ML too.

We had a discussion about this feature, here are some notes:

Use-case:

• Users seem to be mostly interested in numeric fields: metrics, relevance signals, etc.
• We can't assume low volumes of updates, some users seem to be interested in updating a field across most documents of the index on a regular basis for instance.

Implementation:

• We expected most challenges to be on the Lucene side, but the discussion highlighted that they are more on the Elasticsearch side actually, especially to integrate with the _update API and play well with Elasticsearch's replication model. We view the latter as an important requirement.
• These fields should be in the document _source from the user perspective, but because _source doesn't support updating, they shouldn't be stored in the _source stored field but only in doc values and dynamically reintroduced into the _source json document at search time. Might be related to Better storage of _source #9034 which already discusses breaking down the _source into multiple fields.
• We plan to only support single-valued numerics.
• The fact that updates should work with Elasticsearch's replication model probably rules out the idea of maintaining a side-car data-structure. All modifications of the content of the index must be associated with a new document (which may be hidden to users and only used for administrative purposes) with a sequence number. This probably rules out the idea of maintaining a side-car data-structure. However we could use the new soft update API of IndexWriter to atomically update an existing document with a doc-value update and introduce a document whose only purpose is to describe the update operation with a sequence number for replication purposes.

Open questions:

• Would the current implementation of doc values scale well for the workloads of our users? Stacked updates have drawbacks too, they would introduce more complexity on the merging side to keep the number of elements in the stack contained, and would make search slower. We agreed to benchmark to see how it performs for what we think will be common use-cases such as maintaining page view counts.
• How could it work with the _update API? Would it need to refresh to make sure that we have an up-to-date view of the document?
• Probably more questions as we dig further...

Pinging @elastic/es-distributed

anything update for this issue ？

No. We are keeping this idea in the back of our minds, but it is very complex to do it right, and it is not obvious whether it would actually help significantly. For instance we believe it still wouldn't be good enough to index counters.

Sorry for bumping an old thread. Adding three low-volume use cases for which I had to either use parent/child or do bulk updates.

• Tag / untag documents in legal discovery ( document size is very expensive to reindex)
• price updates ( every day) - eCommerce stores
• inventory updates ( every hour )