Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4898

BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.2.0, 2.0.4-alpha
    • 1.3.0, 2.1.1-beta
    • namenode
    • None
    • Reviewed

    Description

      As currently implemented, BlockPlacementPolicyWithNodeGroup does not properly fallback to local rack when no nodes are available in remote racks, resulting in an improper NotEnoughReplicasException.

      BlockPlacementPolicyWithNodeGroup.java
        @Override
        protected void chooseRemoteRack(int numOfReplicas,
            DatanodeDescriptor localMachine, HashMap<Node, Node> excludedNodes,
            long blocksize, int maxReplicasPerRack, List<DatanodeDescriptor> results,
            boolean avoidStaleNodes) throws NotEnoughReplicasException {
          int oldNumOfReplicas = results.size();
          // randomly choose one node from remote racks
          try {
            chooseRandom(
                numOfReplicas,
                "~" + NetworkTopology.getFirstHalf(localMachine.getNetworkLocation()),
                excludedNodes, blocksize, maxReplicasPerRack, results,
                avoidStaleNodes);
          } catch (NotEnoughReplicasException e) {
            chooseRandom(numOfReplicas - (results.size() - oldNumOfReplicas),
                localMachine.getNetworkLocation(), excludedNodes, blocksize,
                maxReplicasPerRack, results, avoidStaleNodes);
          }
        }
      

      As currently coded the chooseRandom() call in the catch block will never succeed as the set of nodes within the passed in node path (e.g. /rack1/nodegroup1) is entirely contained within the set of excluded nodes (both are the set of nodes within the same nodegroup as the node chosen first replica).

      The bug is that the fallback chooseRandom() call in the catch block should be passing in the complement of the node path used in the initial chooseRandom() call in the try block (e.g. /rack1) - namely:

      NetworkTopology.getFirstHalf(localMachine.getNetworkLocation())
      

      This will yield the proper fallback behavior of choosing a random node from within the same rack, but still excluding those nodes in the same nodegroup

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            szetszwo Tsz-wo Sze Assign to me
            sirianni Eric Sirianni
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment