[HDFS-11104] BlockPlacementPolicyDefault choose favoredNodes in turn which may cause imbalance

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha1
• Assignee: Ajith S
• Reporter: Doris Gu
• Created: 04/Nov/16 09:36
• Updated: 30/Oct/17 01:45

Description

if client transfer favoredNodes when it writes files into hdfs,chooseTarget in BlockPlacementPolicyDefault prior chooseTarget in turn:

DatanodeStorageInfo[] chooseTarget(String src,
int numOfReplicas,
Node writer,
Set<Node> excludedNodes,
long blocksize,
List<DatanodeDescriptor> favoredNodes,
BlockStoragePolicy storagePolicy) {
try {
...

for (int i = 0; i < favoredNodes.size() && results.size() < numOfReplicas; i++) {
DatanodeDescriptor favoredNode = favoredNodes.get;
// Choose a single node which is local to favoredNode.
// 'results' is updated within chooseLocalNode
final DatanodeStorageInfo target = chooseLocalStorage(favoredNode,
favoriteAndExcludedNodes, blocksize, maxNodesPerRack,
results, avoidStaleNodes, storageTypes, false);
...

why not shuffle it here? Make block more balanced, save the cost balancer will pay and make cluster more stable.

for (DatanodeDescriptor favoredNode : DFSUtil.shuffle(favoredNodes.toArray(new DatanodeDescriptor[favoredNodes.size()])))

Issue Links

• is related to: HDFS-12561 FavoredNodes should shuffle if more than replicas and make writer first if present at BlockPlacementPolicyDefault#chooseTarget