Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6180

dead node count / listing is very broken in JMX and old GUI

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on the new GUI, but showed up properly in the datanodes tab. Some nodes are also being double reported in the deadnode and inservice section (22 show up dead, 565 show up alive, 9 duplicated nodes).

      From /jmx (confirmed that it's the same in jconsole):

      {
          "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
          "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
          "CapacityTotal" : 5477748687372288,
          "CapacityUsed" : 24825720407,
          "CapacityRemaining" : 5477723861651881,
          "TotalLoad" : 565,
          "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
          "BlocksTotal" : 21065,
          "MaxObjects" : 0,
          "FilesTotal" : 25454,
          "PendingReplicationBlocks" : 0,
          "UnderReplicatedBlocks" : 0,
          "ScheduledReplicationBlocks" : 0,
          "FSState" : "Operational",
          "NumLiveDataNodes" : 565,
          "NumDeadDataNodes" : 0,
          "NumDecomLiveDataNodes" : 0,
          "NumDecomDeadDataNodes" : 0,
          "NumDecommissioningDataNodes" : 0,
          "NumStaleDataNodes" : 1
        },
      

      I'm not going to include deadnode/livenodes because the list is huge, but I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.

        Attachments

        1. HDFS-6180.004.patch
          43 kB
          Haohui Mai
        2. HDFS-6180.003.patch
          43 kB
          Haohui Mai
        3. HDFS-6180.002.patch
          40 kB
          Haohui Mai
        4. HDFS-6180.001.patch
          40 kB
          Haohui Mai
        5. HDFS-6180.000.patch
          35 kB
          Haohui Mai
        6. dn.log
          38 kB
          Travis Thompson

          Issue Links

            Activity

              People

              • Assignee:
                wheat9 Haohui Mai
                Reporter:
                tthompso Travis Thompson
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: