[YARN-4101] RM should print alert messages if Zookeeper and Resourcemanager gets connection issue

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.2, 2.6.2, 3.0.0-alpha1
• Component/s: yarn
• Target Version/s: 2.8.0, 2.7.2, 2.6.2
• Hadoop Flags: Reviewed
• Assignee: Xuan Gong
• Reporter: Yesha Vora
• Created: 01/Sep/15 17:38
• Updated: 06/Jan/17 00:56
• Resolved: 03/Sep/15 01:01

Description

Currently, There is no way for user to understand Zk-RM has connection issues. In HA environment, RM is highly dependent on Zookeeper. If connection between RM and Zk is jeopardized, cluster is likely to be gone in bad state.

Example: Rm1 is active and Rm2 is standby. If connection between Rm2 and Zk is lost, Rm2 will never become active. In this case, if Rm1 hits an error and could not be started, cluster goes in bad state. This situation is very hard to debug for user. In this case, if we can develop better prompting of messages, User could fix the Zk-RM connection issue and could avoid getting in bad state.

Thus, We need a better way to prompt alert to user if connection between Zk -> Active RM or Zk -> standby RM is getting bad.
Here are the suggestions.
1) Print connection lost alert in RM UI
2) Print alert messages while running any Yarn command such as yarn logs, yarn applications etc