[HDFS-8995] Flaw in registration bookeeping can make DN die on reconnect

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.2, 3.0.0-alpha1
• Target Version/s: 2.7.2, 2.6.5
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 31/Aug/15 14:34
• Updated: 06/Jan/17 00:56
• Resolved: 02/Sep/15 01:10

Description

Normally data nodes re-register with the namenode when it was unreachable for more than the heartbeat expiration and becomes reachable again. Datanodes keep retrying the last rpc call such as incremental block report and heartbeat and when it finally gets through the namenode tells it to re-register.

We have observed that some of datanodes stay dead in such scenarios. Further investigation has revealed that those were told to shutdown by the namenode.