[YARN-5694] ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.7.3
• Fix Version/s: 2.6.5, 2.7.4
• Component/s: resourcemanager
• Labels: oct16-medium
• Target Version/s: 2.6.5, 2.7.4
• Hadoop Flags: Reviewed
• Assignee: Daniel Templeton
• Reporter: Daniel Templeton
• Created: 30/Sep/16 19:04
• Updated: 09/Dec/16 07:56
• Resolved: 05/Dec/16 21:50

Description

ZKRMStateStore.doStoreMultiWithRetries() holds the lock while trying to talk to ZK. If the connection fails, it will retry while still holding the lock. The retries are intended to be strictly time limited, but in the case that the ZK node is unreachable, the time limit fails, resulting in the thread holding the lock for over an hour. Transitioning the RM to standby requires that same lock, so in exactly the case that the RM should be transitioning to standby, the VerifyActiveStatusThread blocks it from happening.