[HDFS-16974] Consider volumes average load of each DataNode when choosing target.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: datanode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Shuyan Zhang
• Reporter: Shuyan Zhang
• Created: 09/Apr/23 14:32
• Updated: 28/Jan/24 07:09
• Resolved: 14/Apr/23 02:34

Description

The current target choosing policy only considers the load of the entire datanode. If both DN1 and DN2 have an `xceiverCount` of 100, but DN1 has 10 volumes to write to and DN2 only has 1, then the pressure on DN2 is actually much greater than that on DN1. This patch has added a configuration that allows us to avoid nodes with too much pressure on a single volume when choosing targets, so as to avoid overloading datanodes with few volumes or slowing down writes.

Issue Links

• links to: GitHub Pull Request #5541