Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14920

Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.3, 3.2.1, 3.1.3
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2
    • Component/s: ec
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Decommission test hangs in our clusters.
      Have seen the messages as follow

      2019-10-22 15:58:51,514 TRACE org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Block blk_-9223372035600425840_372987973 numExpected=9, numLive=5
      2019-10-22 15:58:51,514 INFO BlockStateChange: Block: blk_-9223372035600425840_372987973, Expected Replicas: 9, live replicas: 5, corrupt replicas: 0, decommissioned replicas: 0, decommissioning replicas: 4, maintenance replicas: 0, live entering maintenance replicas: 0, excess replicas: 0, Is Open File: false, Datanodes having this block: 10.255.43.57:50010 10.255.53.12:50010 10.255.63.12:50010 10.255.62.39:50010 10.255.37.36:50010 10.255.33.15:50010 10.255.69.29:50010 10.255.51.13:50010 10.255.64.15:50010 , Current Datanode: 10.255.69.29:50010, Is current datanode decommissioning: true, Is current datanode entering maintenance: false
      2019-10-22 15:58:51,514 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager: Node 10.255.69.29:50010 still has 1 blocks to replicate before it is a candidate to finish Decommission In Progress

      After digging the source code and cluster log, guess it happens as follow steps.

      1. Storage strategy is RS-6-3-1024k.
      2. EC block b consists of b0, b1, b2, b3, b4, b5, b6, b7, b8. b0 is from datanode dn0, b1 is from datanode dn1, ...etc
      3. At the beginning dn0 is in decommission progress, b0 is replicated successfully, and dn0 is still in decommission progress.
      4. Later b1, b2, b3 in decommission progress, and dn4 containing b4 is out of service, so need to reconstruct, and create ErasureCodingWork to do it, in the ErasureCodingWork, additionalReplRequired is 4
      5. Because hasAllInternalBlocks is false, Will call ErasureCodingWork#addTaskToDatanode -> DatanodeDescriptor#addBlockToBeErasureCoded, and send BlockECReconstructionInfo task to Datanode
      6. DataNode can not reconstruction the block because targets is 4, greater than 3( parity number).

      There is a problem as follow, from BlockManager.java#scheduleReconstruction

            // should reconstruct all the internal blocks before scheduling
            // replication task for decommissioning node(s).
            if (additionalReplRequired - numReplicas.decommissioning() -
                numReplicas.liveEnteringMaintenanceReplicas() > 0) {
              additionalReplRequired = additionalReplRequired -
                  numReplicas.decommissioning() -
                  numReplicas.liveEnteringMaintenanceReplicas();
            }
      

      Should reconstruction firstly and then replicate for decommissioning. Because numReplicas.decommissioning() is 4, and additionalReplRequired is 4, that's wrong,
      numReplicas.decommissioning() should be 3, it should exclude live replica.
      If so, additionalReplRequired will be 1, reconstruction will schedule as expected. After that, decommission goes on.

        Attachments

        1. HDFS-14920.001.patch
          13 kB
          Hui Fei
        2. HDFS-14920.002.patch
          13 kB
          Hui Fei
        3. HDFS-14920.003.patch
          18 kB
          Hui Fei
        4. HDFS-14920.004.patch
          19 kB
          Hui Fei
        5. HDFS-14920.005.patch
          19 kB
          Hui Fei

          Activity

            People

            • Assignee:
              ferhui Hui Fei
              Reporter:
              ferhui Hui Fei
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: