Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6791

A block could remain under replicated if all of its replicas are on decommissioned nodes

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.6.0
    • None
    • None
    • Reviewed

    Description

      Here is the scenario.

      1. Normally before NN transitions a DN to decommissioned state, enough replicas have been copied to other "in service" DNs. However, in some rare situations, the cluster got into a state where a DN is in decommissioned state and a block's only replica is on that DN. In such state, the number of replication reported by fsck is 1; the block just stays in under replicated state; applications can still read the data, given decommissioned node can served read traffic.

      This can happen in some error situations such DN failure or NN failover. For example
      a) a block's only replica is node A temporarily.
      b) Start decommission process on node A.
      c) When node A is in "decommission-in-progress" state, node A crashed. NN will mark node A as dead.
      d) After node A rejoins the cluster, NN will mark node A as decommissioned.

      2. In theory, NN should take care of under replicated blocks. But it doesn't for this special case where the only replica is on decommissioned node. That is because NN has the policy of "decommissioned node can't be picked the source node for replication".

      BlockManager.java
      chooseSourceDatanode
            // never use already decommissioned nodes
            if(node.isDecommissioned())
              continue;
      

      3. Given NN marks the node as decommissioned, admins will shutdown the datanode. Under replicated blocks turn into missing blocks.

      4. The workaround is to recommission the node so that NN can start the replication from the node.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mingma Ming Ma Assign to me
            mingma Ming Ma
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment