[HDFS-5947] Improve dead node detection and handling

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.0, 2.0.0-alpha, 3.0.0-alpha1
• Component/s: namenode
• Assignee: Unassigned
• Reporter: Daryn Sharp
• Created: 13/Feb/14 16:12
• Updated: 12/May/16 18:16

Description

When HeartbeatManager.heartbeatCheck runs:

1. All DNs are scanned to count dead nodes
2. Processes the first dead node
3. If there was a dead node, loops to re-scan all DNs again

Processing the dead node holds the namesystem write lock while removing the node from the blockmap. It also appears to do a lot of work to immediately re-adjust the replication queues. All this processing might be too expensive while holding the write lock, ex. if a rack or two is lost.