[HDFS-2795] HA: Standby NN takes a long time to recover from a dead DN starting up

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: HA branch (HDFS-1623)
• Fix Version/s: HA branch (HDFS-1623)
• Component/s: datanode, ha, namenode
• Target Version/s: HA branch (HDFS-1623)
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Aaron Myers
• Created: 16/Jan/12 07:46
• Updated: 18/Jan/12 15:35
• Resolved: 17/Jan/12 03:21

Description

To reproduce:

1. Start an HA cluster with a DN.
2. Write several blocks to the FS with replication 1.
3. Shutdown the DN
4. Wait for the NNs to declare the DN dead. All blocks will be under-replicated.
5. Restart the DN.

Note that upon restarting the DN, the active NN will immediately get all block locations from the initial BR. The standby NN will not, and instead will slowly add block locations for a subset of the previously-missing blocks on every DN heartbeat.