Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6180

dead node count / listing is very broken in JMX and old GUI

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 2.3.0
    • 2.5.0
    • None
    • None
    • Reviewed

    Description

      After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on the new GUI, but showed up properly in the datanodes tab. Some nodes are also being double reported in the deadnode and inservice section (22 show up dead, 565 show up alive, 9 duplicated nodes).

      From /jmx (confirmed that it's the same in jconsole):

      {
          "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
          "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
          "CapacityTotal" : 5477748687372288,
          "CapacityUsed" : 24825720407,
          "CapacityRemaining" : 5477723861651881,
          "TotalLoad" : 565,
          "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
          "BlocksTotal" : 21065,
          "MaxObjects" : 0,
          "FilesTotal" : 25454,
          "PendingReplicationBlocks" : 0,
          "UnderReplicatedBlocks" : 0,
          "ScheduledReplicationBlocks" : 0,
          "FSState" : "Operational",
          "NumLiveDataNodes" : 565,
          "NumDeadDataNodes" : 0,
          "NumDecomLiveDataNodes" : 0,
          "NumDecomDeadDataNodes" : 0,
          "NumDecommissioningDataNodes" : 0,
          "NumStaleDataNodes" : 1
        },
      

      I'm not going to include deadnode/livenodes because the list is huge, but I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.

      Attachments

        1. HDFS-6180.004.patch
          43 kB
          Haohui Mai
        2. HDFS-6180.003.patch
          43 kB
          Haohui Mai
        3. HDFS-6180.002.patch
          40 kB
          Haohui Mai
        4. HDFS-6180.001.patch
          40 kB
          Haohui Mai
        5. HDFS-6180.000.patch
          35 kB
          Haohui Mai
        6. dn.log
          38 kB
          Travis Thompson

        Issue Links

          Activity

            People

              wheat9 Haohui Mai
              tthompso Travis Thompson
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: