[OAK-6246] Support for out of band indexing with read only access to NodeStore

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.4, 1.8.0
• Component/s: run
• Epic Link: Index Tooling 1.8
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 19/May/17 14:53
• Updated: 24/Jul/17 13:16
• Resolved: 17/Jul/17 05:56

Description

Provide support for out of band indexing where oak-run is connected in read only mode with NodeStore and indexes are stored on file system. These are then imported back by target system.

Had a discussion with catholicon and following flow was determined

1. Admin would create provision a checkpoint via CheckpointMBean
2. oak-run index is connected to NodeStore in read only mode and passed with
   • checkpoint from previous step
   • list of indexes which need to be reindexed
3. oak-run index logic would then proceed with reindexing. However the created index data would be stored locally. This would make use of
   • DirectoryFactory - OAK-6243
   • Copy-on-write nodestore approach as being used in OAK-6220
4. Once indexing is completed it would dump all index to an output folder with some metadata
5. Then admin can copy this index data and use an MBean on the target setup to "import" it back. This import would need to
   • Pause the current async indexers
   • Import the external index files
   • Bring the external indexer upto date to there respective lanes checkpoint
   • Resume the async indexer

The benefit of this approach is that

1. We only need to backport the import logic. Rest all can be implemented in trunk and need not be backported.
2. Using read-only mode allow oak-run from trunk to be safely connected to any of the old versions

Issue Links

• is blocked by: OAK-6249 FSDirectoryFactory to manage indexes on local file system
• is related to: OAK-5970 (Re-)Indexing: estimate progress / ETA
• is related to: OAK-6353 Use Document order traversal for reindexing performed on DocumentNodeStore setups
• relates to: OAK-6081 Indexing tooling via oak-run

Sub-Tasks

• Support for importing index files: Closed