[HDFS-611] Heartbeats times from Datanodes increase when there are plenty of blocks to delete

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.1, 0.21.0, 0.22.0
• Fix Version/s: 0.21.0, 1.0.0
• Component/s: datanode
• Target Version/s: 1.0.0
• Hadoop Flags: Reviewed
• Assignee: Zheng Shao
• Reporter: Dhruba Borthakur
• Created: 10/Sep/09 08:06
• Updated: 09/Nov/11 21:33
• Resolved: 10/Nov/09 07:40

Description

I am seeing that when we delete a large directory that has plenty of blocks, the heartbeat times from datanodes increase significantly from the normal value of 3 seconds to as large as 50 seconds or so. The heartbeat thread in the Datanode deletes a bunch of blocks sequentially, this causes the heartbeat times to increase.

Issue Links

• is blocked by: HADOOP-6433 Add AsyncDiskService that is used in both hdfs and mapreduce
• is related to: HADOOP-774 Datanodes fails to heartbeat when a directory with a large number of blocks is deleted
• is related to: HADOOP-994 DFS Scalability : a BlockReport that returns large number of blocks-to-be-deleted cause datanode to lost connectivity to namenode
• relates to: MAPREDUCE-1213 TaskTrackers restart is very slow because it deletes distributed cache directory synchronously