[HDFS-7714] Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Component/s: datanode
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Hadoop Flags: Reviewed
• Assignee: Vinayakumar B
• Reporter: Chris Nauroth
• Created: 30/Jan/15 21:05
• Updated: 30/Aug/16 01:39
• Resolved: 10/Feb/15 05:17

Description

In an HA deployment, DataNodes must register with both NameNodes and send periodic heartbeats and block reports to both. However, if NameNodes and DataNodes are restarted simultaneously, then this can trigger a race condition in registration. The end result is that the BPServiceActor for one NameNode terminates, but the BPServiceActor for the other NameNode remains alive. The DataNode process is then in a "half-alive" state where it only heartbeats and sends block reports to one of the NameNodes. This could cause a loss of storage capacity after an HA failover. The DataNode process would have to be restarted to resolve this.

Issue Links

• is related to: HDFS-2882 DN continues to start up, even if block pool fails to initialize
• is related to: HDFS-7009 Active NN and standby NN have different live nodes