[HDFS-4975] Branch-1-win TestReplicationPolicy failed caused by stale data node handling

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1-win
• Fix Version/s: 1-win
• Target Version/s: 1-win
• Hadoop Flags: Reviewed
• Assignee: Douma Fang
• Reporter: Douma Fang
• Created: 09/Jul/13 22:37
• Updated: 16/Jul/13 22:38
• Resolved: 11/Jul/13 20:03

Description

TestReplicationPolicy failed on

• testChooseTargetWithMoreThanAvailableNodes()
• testChooseTargetWithStaleNodes()
• testChooseTargetWithHalfStaleNodes()

The root of cause of testChooseTargetWithMoreThanAvailableNodes failing is the following:
In BlockPlacementPolicyDefault#chooseTarget()

  chooseRandom(numOfReplicas, NodeBase.ROOT, excludedNodes, 
        blocksize, maxNodesPerRack, results);
    } catch (NotEnoughReplicasException e) {
      FSNamesystem.LOG.warn("Not able to place enough replicas, still in need of " + numOfReplicas);

However, numOfReplicas is passed into chooseRandom() as int (primitive type in java) by value. The updating operation for numOfReplicas in chooseRandom() will not change the value in chooseTarget().

The root cause for testChooseTargetWithStaleNodes() and testChooseTargetWithHalfStaleNodes() is the current BlockPlacementPolicyDefault#chooseTarget() doesn't check if a node is stale.

Issue Links

• relates to: HDFS-5001 Branch-1-Win TestAzureBlockPlacementPolicy and TestReplicationPolicyWithNodeGroup failed caused by 1) old APIs and 2) incorrect value of depthOfAllLeaves