[HDFS-14589] RPC fairness for Datanode data transfers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Xue Liu
• Reporter: Íñigo Goiri
• Created: 20/Jun/19 23:51
• Updated: 21/Jun/19 01:47

Description

Currently, the Datanode just replies to the data transfers from the clients as soon as they come.
Eventually, when the DataXceiverServer runs out of threads, it just refuses:

// Make sure the xceiver count is not exceeded
int curXceiverCount = datanode.getXceiverCount();
if (curXceiverCount > maxXceiverCount) {
  throw new IOException("Xceiver count " + curXceiverCount
      + " exceeds the limit of concurrent xcievers: "
      + maxXceiverCount);
}

We had a situation where a user had many containers accessing the same block and ending up saturating the 3 Datanodes and messing with the other users.
Ideally, the Namenode should manage this situation some degree but we can still get into this situation.
We should have some smart in the DN to track this and apply some fairness to the number of requests per user.

Issue Links

• relates to: HDFS-12737 Thousands of sockets lingering in TIME_WAIT state due to frequent file open operations