[OAK-4412] Lucene hybrid index

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.11, 1.6.0
• Component/s: lucene
• Epic Link: indexer resilience 1.6
• Assignee: Chetan Mehrotra
• Reporter: Tomek Rękawek
• Due: 21/Jul/16
• Created: 31/May/16 12:01
• Updated: 08/Oct/19 15:21
• Resolved: 15/Sep/16 07:45

Description

When running Oak in a cluster, each write operation is expensive. After performing some stress-tests with a geo-distributed Mongo cluster, we've found out that updating property indexes is a large part of the overall traffic.

The asynchronous index would be an answer here (as the index update won't be made in the client request thread), but the AEM requires the updates to be visible immediately in order to work properly.

The idea here is to enhance the existing asynchronous Lucene index with a synchronous, locally-stored counterpart that will persist only the data since the last Lucene background reindexing job.

The new index can be stored in memory or (if necessary) in MMAPed local files. Once the "main" Lucene index is being updated, the local index will be purged.

Queries will use an union of results from the lucene and lucene-memory indexes.

The lucene-memory index, as a local stored entity, will be updated using an observer, so it'll get both local and remote changes.

The original idea has been suggested by Chetan Mehrotra in the discussion for the OAK-4233.

Feature Docs
http://jackrabbit.apache.org/oak/docs/query/indexing.html#nrt-indexing

Issue Links

• is blocked by: OAK-4714 Reindexing should only happen in async mode for a mix mode index definition
• is blocked by: OAK-4566 Multiplexing store support in Lucene Indexes
• is blocked by: OAK-4640 Provide a way for commit hook to record meta data for a given commit
• is blocked by: OAK-4641 Using same index definition for both async and sync indexing
• is blocked by: OAK-4753 Add nrt as another index mode
• is related to: OAK-4909 NRTIndex can get closed while in use
• is related to: OAK-6535 Synchronous Lucene Property Indexes
• is related to: OAK-4904 For unique indexes avoid consulting indexes other than property index
• is related to: OAK-5421 Add LuceneDoc directly to queue from LuceneIndexEditor
• is related to: OAK-4809 JMX Stats for NRT Indexing
• is related to: OAK-4906 Lucene: Support relative property based query by transforming the path
• relates to: OAK-4808 Index external changes as part of NRT indexing
• requires: OAK-4767 Provide a way to report additional stats as part of benchmark run
• requires: OAK-4768 Provide an option to enable Metrics collection for benchmark run
• supercedes: OAK-4233 Property index stored locally