[AMBARI-21567] Ambari reports GREEN zookeeper when out of disk space prevents quorum

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.2.0

Environment

Ambari 2.2.0
IBM Open Platform 4.2.0.0
CentOS 6.8

• Assignee: Unassigned
• Reporter: Michael Howard
• Created: 24/Jul/17 20:22
• Updated: 24/Jul/17 20:25

Description

• Ambari reports Zookeeper status as GREEN, even though ZK is not happy and has not reached a quorum.
• Ambari also shows YARN ResourceManager as GREEN, even though it doesn't fully start ... because it is waiting for Zookeeper
• YARN NodeManagers will try to start ... Ambari will report as GREEN and started for a few seconds, but then shows them as stopped.
• YARN NodeManagers are failing to start because ResourceManager is not listening on port 8141 ... because Zookeeper is still waiting for quorum.
• root cause is that one of the Zookeeper nodes ran out of disk space on /opt/hadoop/zookeeper volume ... which prevented a quorum from being formed

Seems like Ambari dashboard should catch the fact that Zookeeper was not in a good state.

In our case, it would have saved significant time in diagnosing the problem.