[OAK-6081] Indexing tooling via oak-run

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.4, 1.8.0
• Component/s: indexing, run
• Epic Link: Index Tooling 1.8
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 17/Apr/17 06:24
• Updated: 08/Oct/19 15:21
• Resolved: 17/Jul/17 09:59

Description

To enable better management for indexing related operation specially around reindexing indexes on large repository setup we should implement some tooling as part of oak-run

The tool would support

1. For DocumentNodeStore setup it would be possible to connect oak-run to a live cluster and it would take care of indexing -> storing index on disk -> merging index -> importing it back at end. This would ensure that live setup faces minimum disruption and is not loaded much
2. For SegementNodeStore setup it would be possible to index on a cloned setup and then provide a way to copy the index back

Future Enhancements

1. Resumable tarversal - It should be able to reindex large repo with resumable traversal such that even if indexing breaks due to some issue it can resume from last state (OAK-5833)
2. Multithreaded traversal - Current indexing is single threaded and hence for large repo it can take long time. Plan here is to support multi threaded indexing where each thread can be assigned a part of repository tree to index and in the end the indexes are merged

Issue Links

• is blocked by: OAK-6117 Enable lucene indexing via oak-run
• is blocked by: OAK-6409 Oak-run indexing: improved (user friendly) output
• is related to: OAK-3680 Partial re-index from last known good state
• is related to: OAK-6246 Support for out of band indexing with read only access to NodeStore
• is related to: OAK-2063 Index creation: interruption resilience
• is related to: OAK-7847 complete "Indexing tooling via oak-run" in 1.8
• relates to: OAK-6453 Script to import oak-run generated indexing to older Oak setup

Sub-Tasks

• Enable lucene indexing via oak-run: Closed
• Enable dumping index definitions and stats via oak-run: Closed
• Enable index consistency check via oak-run: Closed
• Enable dumping index content via oak-run: Closed
• Enable use of pre extracted text cache: Closed
• Implement a DummyDataStore to be used to test setup with no BlobStore access: Closed
• Use BlobStore in read only mode by default unless read-write mode enabled: Closed