[HDFS-6180] dead node count / listing is very broken in JMX and old GUI

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.5.0
• Target Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Haohui Mai
• Reporter: Travis Thompson
• Created: 01/Apr/14 18:10
• Updated: 15/Aug/14 05:41
• Resolved: 07/Apr/14 23:59

Description

After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on the new GUI, but showed up properly in the datanodes tab. Some nodes are also being double reported in the deadnode and inservice section (22 show up dead, 565 show up alive, 9 duplicated nodes).

From /jmx (confirmed that it's the same in jconsole):

{
    "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "CapacityTotal" : 5477748687372288,
    "CapacityUsed" : 24825720407,
    "CapacityRemaining" : 5477723861651881,
    "TotalLoad" : 565,
    "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
    "BlocksTotal" : 21065,
    "MaxObjects" : 0,
    "FilesTotal" : 25454,
    "PendingReplicationBlocks" : 0,
    "UnderReplicatedBlocks" : 0,
    "ScheduledReplicationBlocks" : 0,
    "FSState" : "Operational",
    "NumLiveDataNodes" : 565,
    "NumDeadDataNodes" : 0,
    "NumDecomLiveDataNodes" : 0,
    "NumDecomDeadDataNodes" : 0,
    "NumDecommissioningDataNodes" : 0,
    "NumStaleDataNodes" : 1
  },

I'm not going to include deadnode/livenodes because the list is huge, but I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.

Issue Links

• is broken by: HDFS-3934 duplicative dfs_hosts entries handled wrong
• requires: HDFS-6282 re-add testIncludeByRegistrationName