[HDFS-6706] ZKFailoverController failed to recognize the quorum is not met

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Assignee: Brandon Li
• Reporter: Brandon Li
• Created: 18/Jul/14 21:49
• Updated: 22/Jul/14 02:44
• Resolved: 21/Jul/14 18:51

Description

Thanks Kenny Zhang for finding this problem.
The zkfc cannot be startup due to ha.zookeeper.quorum is not met. "zkfc -format" doesn't log the real problem. And then user will see the error message instead of the real issue when starting zkfc:
2014-07-01 17:08:17,528 FATAL ha.ZKFailoverController (ZKFailoverController.java:doRun(213)) - Unable to start failover controller. Parent znode does not exist.
Run with -formatZK flag to initialize ZooKeeper.

2014-07-01 16:00:48,678 FATAL ha.ZKFailoverController (ZKFailoverController.java:fatalError(365)) - Fatal error occurred:Received create error from Zookeeper. code:NONODE for path /hadoop-ha/prodcluster/ActiveStandbyElectorLock
2014-07-01 17:24:44,202 - INFO ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@627 - Got user-level KeeperException when processing sessionid:0x346f36191250005 type:create cxid:0x2 zxid:0xf00000033 txntype:-1 reqpath:n/a Error Path:/hadoop-ha/prodcluster/ActiveStandbyElectorLock Error:KeeperErrorCode = NodeExists for /hadoop-ha/prodcluster/ActiveStandbyElectorLock

To reproduce the problem:
1. use HDFS cluster with automatic HA enable and set the ha.zookeeper.quorum to 3.
2. start two zookeeper servers.
3. do "hdfs zkfc -format", and then "hdfs zkfc"