[OAK-5528] leaseUpdateThread might be blocked by leaseUpdateCheck

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4, 1.5.14
• Fix Version/s: 1.4.15, 1.6.0
• Component/s: core
• Epic Link: DocumentMK resilience
• Assignee: Julian Reschke
• Reporter: Stefan Eissing
• Created: 26/Jan/17 16:54
• Updated: 16/Feb/21 09:52
• Resolved: 26/Jan/17 17:03

Description

cloned from OAK-5446 due to internal JIRA issues

Fighting with cluster nodes losing their lease and shutting down oak-core in a cloud environment. For reasons unknown at this point in time, the whole process seems to skip about two minutes of real time.

This is a situation from which oak currently does not recover. Code analysis shows that ClusterNodeInfo is handed the LeaseCheckDocumentStoreWrapper instance to use as store. This is fatal since any action the renewLease() tries to do will first invoke the performLeaseCheck(). The lease check will, when the FailureMargin is reached, stall the renewLease() thread for 5 retry attempts and then declare the lease to be lost.

The ClusterNodeInfo should instead be using the "real" DocumentStore, not the wrapped one, IMO.

Issue Links

• is a clone of: OAK-5446 leaseUpdateThread might be blocked by leaseUpdateCheck
• is related to: OAK-3399 5sec retry loop before declaring lease failure (was: as a last resort try to update it there)
• is related to: OAK-5523 LeaseUpdateRetryLoop: add test coverage