[HDFS-8277] Safemode enter fails when Standby NameNode is down

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Component/s: ha, namenode

Environment

HDP 2.2.0

• Assignee: Jianfei Jiang
• Reporter: Hari Sekhon
• Created: 28/Apr/15 15:09
• Updated: 18/Mar/22 03:37

Description

HDFS fails to enter safemode when the Standby NameNode is down (eg. due to AMBARI-10536).

hdfs dfsadmin -safemode enter
safemode: Call From nn2/x.x.x.x to nn1:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

This appears to be a bug in that it's not trying both NameNodes like the standard hdfs client code does, and is instead stopping after getting a connection refused from nn1 which is down. I verified normal hadoop fs writes and reads via cli did work at this time, using nn2. I happened to run this command as the hdfs user on nn2 which was the surviving Active NameNode.

After I re-bootstrapped the Standby NN to fix it the command worked as expected again.

Issue Links

• is duplicated by: HDFS-15509 Set safemode should not fail if one of the namenode is down.
• relates to: HDFS-6507 Improve DFSAdmin to support HA cluster better