[OAK-4808] Index external changes as part of NRT indexing

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.18, 1.6.0
• Component/s: lucene
• Assignee: Chetan Mehrotra
• Reporter: Chetan Mehrotra
• Created: 15/Sep/16 06:37
• Updated: 25/Apr/22 13:34
• Resolved: 11/Jan/17 05:41

Description

With OAK-4412 NRT indexing support in hybrid index case indexes local changes. It would be useful to have an option to index external changes. In ideal world

1. Async indexing is configured at 5 sec internal
2. DocumentNodeStore background read is configured at 1 sec interval

So we can index 5 external changes before changes via async indexing are picked up. In real world delays can happen in both parts so having such changes indexed via NRT mode would be useful to reduce the latency in reflecting external changes as part of query result.

This part would though introduce a cost as it would do complete diff for each external change. So while implementing care must be taken to do it on best effort basis i.e. if queue becomes large then skip certain processing etc

Issue Links

• is related to: OAK-9751 Exception while reading external changes from journal
• is related to: OAK-5177 Non default MissingIndexProviderStrategy is not being passed to child editor
• is related to: OAK-6513 Journal based Async Indexer
• is related to: OAK-4412 Lucene hybrid index
• is related to: OAK-3573 BackgroundObserver should decide about collapsing external events based on backend
• is related to: OAK-5101 Enable prefiltering for external changes
• relates to: OAK-5434 Skip processing of queued changes if async index update is detected in ExternalIndexObserver