[HADOOP-10722] Standby NN continuing as standby when active NN machine got shutdown.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 2.4.0
• Component/s: auto-failover, ha
• Assignee: Unassigned
• Reporter: Surendra Singh Lilhore
• Created: 19/Jun/14 09:59
• Updated: 19/Jun/14 10:55
• Resolved: 19/Jun/14 10:55

Description

I have HA cluster with 3 ZK, 3 QJM.
My Active NN machine got shutdown, but still my standby NN is standby only.
It should be active

ZKFC logs
========

2014-06-19 13:39:30,810 INFO org.apache.hadoop.ha.NodeFencer: ====== Beginning Service Fencing Process... ======
2014-06-19 13:39:30,810 INFO org.apache.hadoop.ha.NodeFencer: Trying method 1/1: org.apache.hadoop.ha.SshFenceByTcpPort(null)
2014-06-19 13:39:30,811 INFO org.apache.hadoop.ha.SshFenceByTcpPort: Connecting to host-10-18-40-101...
2014-06-19 13:39:30,811 INFO org.apache.hadoop.ha.SshFenceByTcpPort.jsch: Connecting to host-10-18-40-101 port 22
2014-06-19 13:39:33,814 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to connect to host-10-18-40-101 as user myuser
com.jcraft.jsch.JSchException: java.net.NoRouteToHostException: No route to host
	at com.jcraft.jsch.Util.createSocket(Util.java:386)
	at com.jcraft.jsch.Session.connect(Session.java:182)
	at org.apache.hadoop.ha.SshFenceByTcpPort.tryFence(SshFenceByTcpPort.java:100)
	at org.apache.hadoop.ha.NodeFencer.fence(NodeFencer.java:97)
	at org.apache.hadoop.ha.ZKFailoverController.doFence(ZKFailoverController.java:521)
	at org.apache.hadoop.ha.ZKFailoverController.fenceOldActive(ZKFailoverController.java:494)
	at org.apache.hadoop.ha.ZKFailoverController.access$1100(ZKFailoverController.java:59)
	at org.apache.hadoop.ha.ZKFailoverController$ElectorCallbacks.fenceOldActive(ZKFailoverController.java:837)
	at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:901)
	at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:800)
	at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:415)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:596)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
2014-06-19 13:39:33,814 WARN org.apache.hadoop.ha.NodeFencer: Fencing method org.apache.hadoop.ha.SshFenceByTcpPort(null) was unsuccessful.