[HDFS-9239] DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: datanode, namenode
• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed

Release Note

This release adds a new feature called the DataNode Lifeline Protocol. If configured, then DataNodes can report that they are still alive to the NameNode via a fallback protocol, separate from the existing heartbeat messages. This can prevent the NameNode from incorrectly marking DataNodes as stale or dead in highly overloaded clusters where heartbeat processing is suffering delays. For more information, please refer to the hdfs-default.xml documentation for several new configuration properties: dfs.namenode.lifeline.rpc-address, dfs.namenode.lifeline.rpc-bind-host, dfs.datanode.lifeline.interval.seconds, dfs.namenode.lifeline.handler.ratio and dfs.namenode.lifeline.handler.count.

• Assignee: Chris Nauroth
• Reporter: Chris Nauroth
• Created: 14/Oct/15 04:20
• Updated: 21/May/19 02:49
• Resolved: 04/Mar/16 23:44

Description

This issue proposes introduction of a new feature: the DataNode Lifeline Protocol. This is an RPC protocol that is responsible for reporting liveness and basic health information about a DataNode to a NameNode. Compared to the existing heartbeat messages, it is lightweight and not prone to resource contention problems that can harm accurate tracking of DataNode liveness currently. The attached design document contains more details.

Issue Links

• relates to: HDFS-7060 Avoid taking locks when sending heartbeats from the DataNode
• relates to: HDFS-9311 Support optional offload of NameNode HA service health checks to a separate RPC server.