[OAK-4668] Make async index more resilient on documentmk

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.9, 1.6.0, 1.4.16
• Component/s: query
• Epic Link: indexer resilience 1.6
• Assignee: Alex Deparvu
• Reporter: Alex Deparvu
• Created: 11/Aug/16 13:23
• Updated: 16/Feb/21 09:56
• Resolved: 16/Aug/16 09:22

Description

The async index update suffers in an eventual consistency store from stale reads of the root node. It may happen that in a cluster an indexing job running concurrently on more nodes, would not see the lease updates coming from another cluster node, so it would end up wiping away checkpoints and/or triggering a full reindex.
There is not much we can do at this level, but make the code a bit more resilient for this specific case (reindex due to missing reference checkpoint) by issuing a write+read operation to force a root revision update which would hopefully prevent a full reindex.
For background, the preferred solution is to have an upper layer choose the node where the async jobs run (aka. use sling discovery bundles).

Issue Links

• is related to: OAK-4734 AsyncIndexUpdateClusterTestIT fails occasionally