Closed
Description
Today when users restore a snapshot Elasticsearch creates an index by copying the contents of the repository onto disk, and only starts each shard once the copy is complete. However in many cases we do not need to copy the whole shard to disk before being able to search it. This meta-issue tracks the work towards an alternative restore mode which works lazily, starting the shard up after restoring only a fraction of the data and restoring the remaining data in the background and/or as needed to serve searches.
- Add plugin infrastructure (Add Searchable Snapshots plugin project #50022)Support searching a shard that does not really exist on disk (Add Lucene directory and index input implementations that expose shard snapshot #49651)Support restoring an index without copying any data (Add SearchableSnapshotRepository #50239)Add disk-based caching layer to avoid duplicate work (Add searchable snapshots cache directory #50693)Associate translog with last Lucene index commit for searchable snapshots shards (Associate translog with Lucene index commit for searchable snapshots shards #53459)Add instrumentation (Add new Lucene directory to track Lucene files read access usages #50283)Support restoring from S3 (Add ranged readBlob to S3BlobContainer #51137)Support freezing of searchable snapshots (Allow freezing searchable snapshots #52653)Support restoring from GCS (Add GCS support for searchable snapshots #55403)Support restoring from Azure (Add Azure support for ranged read blob operations #54358)Support other backends TBDAutomatically reallocate and redo a restore if the primary is lost (Auto-allocate searchable snapshots #52527)Think about how to snapshot a cluster containing searchable snapshots (e.g. expunge relevant settings?); should a restore reinstate the searchable snapshots too?Disable readahead on reads that bypass the cache (via
readDirectly()
)Revisit UX decisions, particularly around needing a separate repository. Can we make the laziness a property of the individual restore instead?superseded by Add API to mount a snapshot as a searchable index #53084Gracefully handle attempts to remove a repository containing mounted snapshots.Documentation (concepts, APIs, ...)HLRC integration
Metadata
Metadata
Assignees
Type
Projects
Milestone
Relationships
Development
No branches or pull requests
Activity
Add stats for time spent fetching data while searching snapshots (#51866
Add Clear Cache API for Searchable Snapshots (#53009)
Add API to mount a snapshot as a searchable index
Add API to mount a snapshot as a searchable index (#53084)
38 remaining items