[HDFS-9318] considerLoad factor can be improved

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Kuhu Shukla
• Reporter: Kuhu Shukla
• Created: 27/Oct/15 19:25
• Updated: 12/May/16 18:19
• Resolved: 06/Nov/15 20:09

Description

Currently considerLoad avoids choosing nodes that are too active, so it helps level the HDFS load across the cluster. Under normal conditions, this is desired. However, when a cluster has a large percentage of nearly full nodes, this can make it difficult to find good targets because the placement policy wants to avoid the full nodes, but considerLoad wants to avoid the busy less-full nodes.