[HDFS-14648] Implement DeadNodeDetector basic model

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0
• Hadoop Flags: Reviewed
• Assignee: Lisheng Sun
• Reporter: Lisheng Sun
• Created: 15/Jul/19 13:12
• Updated: 16/Nov/19 03:48
• Resolved: 16/Nov/19 03:37

Description

This Jira constructs DeadNodeDetector state machine model. The function it implements as follow:

1. When a DFSInputstream is opened, a BlockReader is opened. If some DataNode of the block is found to inaccessible, put the DataNode into DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when DataNode is not accessible, it is likely that the replica has been removed from the DataNode.Therefore, it needs to be confirmed by re-probing and requires a higher priority processing.
2. DeadNodeDetector will periodically detect the Node in DeadNodeDetector#deadnode, If the access is successful, the Node will be moved from DeadNodeDetector#deadnode. Continuous detection of the dead node is necessary. The DataNode need rejoin the cluster due to a service restart/machine repair. The DataNode may be permanently excluded if there is no added probe mechanism.
3. DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using DataNode. When the DFSInputstream is closed, it will be moved from DeadNodeDetector#dfsInputStreamNodes.
4. Every time get the global deanode, update the DeadNodeDetector#deadnode. The new DeadNodeDetector#deadnode Equals to the intersection of the old DeadNodeDetector#deadnode and the Datanodes are by DeadNodeDetector#dfsInputStreamNodes.
5. DeadNodeDetector has a switch that is turned off by default. When it is closed, each DFSInputstream still uses its own local deadnode.
6. This feature has been used in the XIAOMI production environment for a long time. Reduced hbase read stuck, due to node hangs.
7. Just open the DeadNodeDetector switch and you can use it directly. No other restrictions. Don't want to use DeadNodeDetector, just close it.

   if (sharedDeadNodesEnabled && deadNodeDetector == null) {
     deadNodeDetector = new DeadNodeDetector(name);
     deadNodeDetectorThr = new Daemon(deadNodeDetector);
     deadNodeDetectorThr.start();
   }
   

Issue Links

• links to: GitHub Pull Request #1309