[HDFS-3703] Decrease the datanode failure detection time

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.3, 2.0.0-alpha, 3.0.0-alpha1
• Fix Version/s: 1.1.0, 2.0.3-alpha
• Component/s: datanode, namenode
• Hadoop Flags: Reviewed

Release Note

This jira adds a new DataNode state called "stale" at the NameNode. DataNodes are marked as stale if it does not send heartbeat message to NameNode within the timeout configured using the configuration parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds). NameNode picks a stale datanode as the last target to read from when returning block locations for reads.

This feature is by default turned * off *. To turn on the feature, set the HDFS configuration "dfs.namenode.check.stale.datanode" to true.

• Assignee: Jing Zhao
• Reporter: Nicolas Liochon
• Created: 23/Jul/12 11:02
• Updated: 12/May/16 18:13
• Resolved: 14/Sep/12 12:36

Description

By default, if a box dies, the datanode will be marked as dead by the namenode after 10:30 minutes. In the meantime, this datanode will still be proposed by the nanenode to write blocks or to read replicas. It happens as well if the datanode crashes: there is no shutdown hooks to tell the nanemode we're not there anymore.
It especially an issue with HBase. HBase regionserver timeout for production is often 30s. So with these configs, when a box dies HBase starts to recover after 30s and, while 10 minutes, the namenode will consider the blocks on the same box as available. Beyond the write errors, this will trigger a lot of missed reads:

• during the recovery, HBase needs to read the blocks used on the dead box (the ones in the 'HBase Write-Ahead-Log')
• after the recovery, reading these data blocks (the 'HBase region') will fail 33% of the time with the default number of replica, slowering the data access, especially when the errors are socket timeout (i.e. around 60s most of the time).

Globally, it would be ideal if HDFS settings could be under HBase settings.
As a side note, HBase relies on ZooKeeper to detect regionservers issues.

Issue Links

• is related to: HDFS-4350 Make enabling of stale marking on read and write paths independent
• is related to: HDFS-1599 Umbrella Jira for Improving HBASE support in HDFS
• is related to: HBASE-6751 Too many retries, leading a a delay to read the HLog after a datanode failure
• relates to: HDFS-3912 Detecting and avoiding stale datanodes for writing
• relates to: HBASE-5843 Improve HBase MTTR - Mean Time To Recover

Sub-Tasks

• Detecting and avoiding stale datanodes for writing: Closed
• Add number of stale DataNodes to metrics: Closed
• Add number of stale DataNodes to metrics for Branch-1: Closed