Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6180

dead node count / listing is very broken in JMX and old GUI

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on the new GUI, but showed up properly in the datanodes tab. Some nodes are also being double reported in the deadnode and inservice section (22 show up dead, 565 show up alive, 9 duplicated nodes).

      From /jmx (confirmed that it's the same in jconsole):

      {
          "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
          "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
          "CapacityTotal" : 5477748687372288,
          "CapacityUsed" : 24825720407,
          "CapacityRemaining" : 5477723861651881,
          "TotalLoad" : 565,
          "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
          "BlocksTotal" : 21065,
          "MaxObjects" : 0,
          "FilesTotal" : 25454,
          "PendingReplicationBlocks" : 0,
          "UnderReplicatedBlocks" : 0,
          "ScheduledReplicationBlocks" : 0,
          "FSState" : "Operational",
          "NumLiveDataNodes" : 565,
          "NumDeadDataNodes" : 0,
          "NumDecomLiveDataNodes" : 0,
          "NumDecomDeadDataNodes" : 0,
          "NumDecommissioningDataNodes" : 0,
          "NumStaleDataNodes" : 1
        },
      

      I'm not going to include deadnode/livenodes because the list is huge, but I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.

      1. HDFS-6180.004.patch
        43 kB
        Haohui Mai
      2. HDFS-6180.003.patch
        43 kB
        Haohui Mai
      3. HDFS-6180.002.patch
        40 kB
        Haohui Mai
      4. HDFS-6180.001.patch
        40 kB
        Haohui Mai
      5. HDFS-6180.000.patch
        35 kB
        Haohui Mai
      6. dn.log
        38 kB
        Travis Thompson

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Haohui Mai
              Reporter:
              Travis Thompson
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development