[OAK-2682] Introduce time difference detection for DocumentNodeStore

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.4, 1.4
• Component/s: core, mongomk, rdbmk
• Labels: resilience
• Assignee: Stefan Egli
• Reporter: Stefan Egli
• Created: 25/Mar/15 16:56
• Updated: 26/Sep/19 11:43
• Resolved: 12/Aug/15 12:33

Description

Currently the lease mechanism in DocumentNodeStore/mongoMk is based on the assumption that the clocks are in perfect sync between all nodes of the cluster. The lease is valid for 60sec with a timeout of 30sec. If clocks are off by too much, and background operations happen to take couple seconds, you run the risk of timing out a lease. So introducing a check which WARNs if the clocks in a cluster are off by too much (1st threshold, eg 5sec?) would help increase awareness. Further drastic measure could be to prevent a startup of Oak at all if the difference is for example higher than a 2nd threshold (optional I guess, but could be 20sec?).

Issue Links

• blocks: OAK-2844 Introducing a simple document-based discovery-light service (to circumvent documentMk's eventual consistency delays)
• blocks: SLING-4603 discovery.oak: oak-based discovery implementation
• blocks: OAK-2739 take appropriate action when lease cannot be renewed (in time)
• is related to: OAK-4984 Server time unavailable with authenticated connection to MongoDB
• is related to: OAK-3623 backport DocumentStore time diff API back
• relates to: OAK-2681 Update lease without holding lock
• relates to: OAK-3238 fine tune clock-sync check vs lease-check settings