[HDFS-15809] DeadNodeDetector doesn't remove live nodes from dead node set.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1, 3.4.0
• Fix Version/s: 3.3.1, 3.4.0
• Component/s: datanode
• Target Version/s: 3.3.1, 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Jinglun
• Reporter: Jinglun
• Created: 01/Feb/21 13:38
• Updated: 11/Feb/24 13:53
• Resolved: 15/Mar/21 03:39

Description

We found the dead node detector might never remove the alive nodes from the dead node set in a big cluster. For example:

1. 200 nodes are added to the dead node set by DeadNodeDetector.
2. DeadNodeDetector#checkDeadNodes() adds 100 nodes to the deadNodesProbeQueue because the queue limited length is 100.
3. The probe threads start working and probe 30 nodes.
4. DeadNodeDetector#checkDeadNodes() is scheduled again. It iterates the dead node set  and adds 30 nodes to the deadNodesProbeQueue. But the order is the same as the last time. So the 30 nodes that has already been probed are added to the queue again.
5. Repeat 3 and 4. But we always add the first 30 nodes from the dead set. If they are all dead then the live nodes behind them could never be recovered.

Issue Links

• relates to: HDFS-13571 Deadnode detection