[HDFS-15443] Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: datanode
• Assignee: AMC-team
• Reporter: AMC-team
• Created: 27/Jun/20 17:52
• Updated: 29/Apr/22 06:00
• Resolved: 08/Aug/20 09:15

Description

Configuration parameter dfs.datanode.max.transfer.threads is to specify the maximum number of threads to use for transferring data in and out of the DN. This is a vital param that need to tune carefully. 

// DataXceiverServer.java
// Make sure the xceiver count is not exceeded
intcurXceiverCount = datanode.getXceiverCount();
if (curXceiverCount > maxXceiverCount) {
thrownewIOException("Xceiver count " + curXceiverCount
+ " exceeds the limit of concurrent xceivers: "
+ maxXceiverCount);
}

There are many issues that caused by not setting this param to an appropriate value. However, there is no any check code to restrict the parameter. Although having a hard-and-fast rule is difficult because we need to consider number of cores, main memory etc, we can prevent users from setting this value to an absolute wrong value by accident. (e.g. a negative value that totally break the availability of datanode.)

How to fix:

Add proper check code for the parameter.