[HDFS-6397] NN shows inconsistent value in deadnode count

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.4.1
• Target Version/s: 2.5.0, 2.4.1
• Hadoop Flags: Reviewed
• Assignee: Mohammad Islam
• Reporter: Mohammad Islam
• Created: 14/May/14 04:33
• Updated: 04/Sep/14 01:11
• Resolved: 19/May/14 18:13

Description

Context:
When NN is started , without any live datanode but there are nodes in the dfs.includes, NN shows the deadcount as '0'.

There are two inconsistencies:
1. If you click on deadnode links (which shows the count is 0), it will display the list of deadnodes correctly.
2. hadoop 1.x used to display the count correctly.

The following snippets of JMX response will explain it further:
Look at the value of "NumDeadDataNodes"

 {
    "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "CapacityTotal" : 0,
    "CapacityUsed" : 0,
... 
   "NumLiveDataNodes" : 0,
    "NumDeadDataNodes" : 0,
    "NumDecomLiveDataNodes" : 0,
    "NumDecomDeadDataNodes" : 0,
    "NumDecommissioningDataNodes" : 0,
    "NumStaleDataNodes" : 0
  },

Look at " "DeadNodes"".

{
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    
....
    "TotalBlocks" : 70,
    "TotalFiles" : 129,
    "NumberOfMissingBlocks" : 0,
    "LiveNodes" : "{}",
    "DeadNodes" : "{\"<MMMMM>.linkedin.com\":{\"lastContact\":1400037397,\"decommissioned\":false,\"xferaddr\":\"172.XX.X.XX:71\"},\"<NNNNN>.linkedin.com\":{\"lastContact\":1400037397,\"decommissioned\":false,\"xferaddr\":\"172.XX.XX.XX:71\"}}",
    "DecomNodes" : "{}",
   .....
  }