[HDFS-15896] Add used percent limitation to BlockPlacementPolicyDefault for choosing DataNode to write

• Type: Improvement
• Status: Patch Available
• Priority: Minor
• Resolution: Unresolved
• Component/s: block placement

Environment

• Assignee: Yang Yun
• Reporter: Yang Yun
• Created: 15/Mar/21 06:59
• Updated: 16/Mar/21 12:45

Description

Add used percent limitation to BlockPlacementPolicyDefault for choosing datanode to write.

The logical is similar with avoid stale node.

Default is disable, the high used percent is 100.0%

public static final String 
 DFS_NAMENODE_HIGH_USED_DATANODE_FOR_WRITE_PERCENT_KEY =
 "dfs.namenode.high-used.datanode.for.wirte.percent";
public static final float
 DFS_NAMENODE_HIGH_USED_DATANODE_FOR_WRITE_PERCENT_DEFAULT =
 100.0f;

The choosing will try two times, if the first failed for high used datanode, it will try again without high used limitation.

Also add a high  used ratio,  when the percentage of high used datanodes reaches this ratio, allow writing to high used nodes to prevent hotspots.

public static final String
 DFS_NAMENODE_HIGH_USED_DATANODE_FOR_WRITE_RATIO_KEY =
 "dfs.namenode.high-used.datanode.for.wirte.ratio";
public static final float
 DFS_NAMENODE_HIGH_USED_DATANODE_FOR_WRITE_RATIO_DEFAULT = 0.6f;