[OAK-4859] Warn if lease update is invoked with large delay

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.10
• Fix Version/s: 1.4.15, 1.5.13, 1.6.0, 1.2.26, 1.0.42
• Component/s: documentmk
• Epic Link: DocumentMK resilience
• Assignee: Stefan Egli
• Reporter: Stefan Egli
• Created: 28/Sep/16 11:59
• Updated: 29/Aug/19 11:25
• Resolved: 01/Nov/16 14:51

Description

DocumentMk's lease mechanism is built on the assumption that the lease is periodically updated by each instance. If the update doesn't happen within a certain time - and the instance hasn't crashed - there's the risk of the own lease to fail. It is therefore important that the lease update happens without (large) delay according to the configured period.
One pattern where this doesn't happen is when the VM is under heavy load due to JVM-Full-GC cycles. It seems likely that a memory problem doesn't normally happen instantly, but slowly builds up. Based on this assumption we could introduce a check that compares the actual time since last lease update with the desired period. If these two diverge a lot then we can at least issue a log.warn. This might help to easier identify this type of lease failures and perhaps find root causes earlier/easier.

Issue Links

• is related to: OAK-7272 improve BackgroundLeaseUpdate warning messages