[HDFS-4721] Speed up lease/block recovery when DN fails and a block goes into recovery

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.3-alpha
• Fix Version/s: 2.1.0-beta
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Varun Sharma
• Reporter: Varun Sharma
• Created: 21/Apr/13 17:17
• Updated: 27/Aug/13 22:08
• Resolved: 26/Apr/13 20:53

Description

This was observed while doing HBase WAL recovery. HBase uses append to write to its write ahead log. So initially the pipeline is setup as

DN1 --> DN2 --> DN3

This WAL needs to be read when DN1 fails since it houses the HBase regionserver for the WAL.

HBase first recovers the lease on the WAL file. During recovery, we choose DN1 as the primary DN to do the recovery even though DN1 has failed and is not heartbeating any more.

Avoiding the stale DN1 would speed up recovery and reduce hbase MTTR. There are two options.
a) Ride on HDFS 3703 and if stale node detection is turned on, we do not choose stale datanodes (typically not heart beated for 20-30 seconds) as primary DN(s)
b) We sort the replicas in order of last heart beat and always pick the ones which gave the most recent heart beat

Going to the dead datanode increases lease + block recovery since the block goes into UNDER_RECOVERY state even though no one is recovering it actively. Please let me know if this makes sense. If yes, whether we should move forward with a) or b).

Thanks

Issue Links

• is related to: HDFS-4796 Port HDFS-4721 'Speed up lease/block recovery when DN fails and a block goes into recovery' to branch 1
• is required by: HBASE-5843 Improve HBase MTTR - Mean Time To Recover
• relates to: HBASE-8389 HBASE-8354 forces Namenode into loop with lease recovery requests
• relates to: HDFS-4724 Provide API for checking whether lease is recovered or not
• relates to: HDFS-4754 Add an API in the namenode to mark a datanode as stale