[HDFS-17484] Introduce redundancy.considerLoad.minLoad to avoiding excluding nodes when they are not busy actually

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: namenode
• Labels: pull-request-available
• Assignee: farmmamba
• Reporter: farmmamba
• Created: 22/Apr/24 05:59
• Updated: 29/Apr/24 01:49

Description

Currently, we have `dfs.namenode.redundancy.considerLoad` equals true by default, and 

dfs.namenode.redundancy.considerLoad.factor equals 2.0 by default.

Think about below situation. when we are doing stress test, we may deploy hdfs client onto the datanode. So, this hdfs client will prefer to write to its local datanode and increase this machine's load.  Suppose we have 3 datanodes, the load of them are as below:  5.0, 0.2, 0.3.

 

The load equals to 5.0 will be excluded when choose datanodes for a block. But actually, it is not slow node when load equals to 5.0 for a machine with 80 cpu cores.

 

So, we should better add a new configuration entry :  `dfs.namenode.redundancy.considerLoad.minLoad` to indicate the mininum factor we will make considerLoad take effect.

Issue Links

• links to: GitHub Pull Request #6758