[HDFS-5922] DN heartbeat thread can get stuck in tight loop

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: datanode
• Target Version/s: 2.4.0
• Hadoop Flags: Reviewed
• Assignee: Arpit Agarwal
• Reporter: Aaron Myers
• Created: 10/Feb/14 22:20
• Updated: 04/Sep/14 01:15
• Resolved: 25/Feb/14 02:28

Description

We saw an issue recently on a test cluster where one of the DN threads was consuming 100% of a single CPU. Running jstack indicated that it was the DN heartbeat thread. I believe I've tracked down the cause to a bug in the accounting around the value of pendingReceivedRequests.

More details in the first comment.