[OAK-8627] Avoid late-arriving lastRev update from crashed instance

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0
• Fix Version/s: 1.18.0
• Component/s: documentmk
• Assignee: Marcel Reutegger
• Reporter: Stefan Egli
• Created: 16/Sep/19 10:31
• Updated: 27/Apr/20 13:02
• Resolved: 24/Sep/19 13:15

Description

Recently a deployment with a two node cluster showed a Sling Discovery Oak with a cluster view that had a clusterId stuck in the deactivating state.

According to the entry in the clusterNodes collection, the clusterId in the deactivating state was inactive. However, the revisions for the _lastRev entry on the root document and the lastWrittenRootRev did not match. The latter was slightly more recent. This caused the Sling Discovery Oak to consider the clusterId as not entirely shut down.

While there is no direct proof, one theoretical scenario mreutegg identified as a potential root cause was that it can happen that the lastRev for a clusterId on the root document is set back to an earlier value due to a race condition:

Before the lease expiry, the background update thread could have issued an update for the root document, which then took a very long time to reach the DocumentStore, longer than the lease timeout and recovery which must have been done by another instance meanwhile.

If such a late-arriving update of the _lastRev is possible, then the reset of the lastRev value on the root document could be explained, since the update is currently done unconditionally.