Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6180

dead node count / listing is very broken in JMX and old GUI

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After bringing up a 578 node cluster with 13 dead nodes, 0 were reported on the new GUI, but showed up properly in the datanodes tab. Some nodes are also being double reported in the deadnode and inservice section (22 show up dead, 565 show up alive, 9 duplicated nodes).

      From /jmx (confirmed that it's the same in jconsole):

      {
          "name" : "Hadoop:service=NameNode,name=FSNamesystemState",
          "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
          "CapacityTotal" : 5477748687372288,
          "CapacityUsed" : 24825720407,
          "CapacityRemaining" : 5477723861651881,
          "TotalLoad" : 565,
          "SnapshotStats" : "{\"SnapshottableDirectories\":0,\"Snapshots\":0}",
          "BlocksTotal" : 21065,
          "MaxObjects" : 0,
          "FilesTotal" : 25454,
          "PendingReplicationBlocks" : 0,
          "UnderReplicatedBlocks" : 0,
          "ScheduledReplicationBlocks" : 0,
          "FSState" : "Operational",
          "NumLiveDataNodes" : 565,
          "NumDeadDataNodes" : 0,
          "NumDecomLiveDataNodes" : 0,
          "NumDecomDeadDataNodes" : 0,
          "NumDecommissioningDataNodes" : 0,
          "NumStaleDataNodes" : 1
        },
      

      I'm not going to include deadnode/livenodes because the list is huge, but I've confirmed there are 9 nodes showing up in both deadnodes and livenodes.

        Attachments

        1. dn.log
          38 kB
          Travis Thompson
        2. HDFS-6180.000.patch
          35 kB
          Haohui Mai
        3. HDFS-6180.001.patch
          40 kB
          Haohui Mai
        4. HDFS-6180.002.patch
          40 kB
          Haohui Mai
        5. HDFS-6180.003.patch
          43 kB
          Haohui Mai
        6. HDFS-6180.004.patch
          43 kB
          Haohui Mai

        Issue Links

          Activity

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              tthompso Travis Thompson

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment