[HDFS-15636] NameNode computes load by group when choosing datanodes.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Jinglun
• Reporter: Jinglun
• Created: 16/Oct/20 07:13
• Updated: 16/Oct/20 20:44
• Resolved: 16/Oct/20 07:49

Description

We have an HDFS cluster used for HBase with 251 ssd datanodes and 30 hdd datanodes. The HOT files are stored with ALL_SSD and cold ones are stored with HOT. There is a big chance the NameNode couldn't choose enough nodes for writing disk files(with storage policy HOT) because of 'NODE_TOO_BUSY'. A temporary solution is  to increase the 'dfs.namenode.redundancy.considerLoad.factor'. But that may cause the unbalance of load of all the datanodes.
We should let the NameNode compute load by group. The ssd nodes and hdd nodes are computed separately and each group has its own average load. When the NameNode chooses a hdd node it only compares the node's load with
the average load of the hdd group.

Issue Links

• duplicates: HDFS-14383 Compute datanode load based on StoragePolicy