[HADOOP-774] Datanodes fails to heartbeat when a directory with a large number of blocks is deleted

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Assignee: Dhruba Borthakur
• Reporter: Dhruba Borthakur
• Created: 02/Dec/06 00:28
• Updated: 27/Oct/09 01:27
• Resolved: 07/Dec/06 19:35

Description

If a user removes a few files that are huge, it causes the namenode to send BlockInvalidate command to the relevant Datanodes. The Datanode process the blockInvalidate command as part of its heartbeat thread. If the number of blocks to be invalidated is huge, the datanode takes a long time to process it. This causes the datanode to not send new heartbeats to the namenode. The namenode declares the datanode as dead!

1. One option is to process the blockInvalidate as a separate thread from the heartbeat thread in the Datanode.
2. Another option would be to constrain the namenode to send a max (e.g. 500) blocks per blockInvalidate message.

Issue Links

• relates to: HADOOP-994 DFS Scalability : a BlockReport that returns large number of blocks-to-be-deleted cause datanode to lost connectivity to namenode
• relates to: HDFS-611 Heartbeats times from Datanodes increase when there are plenty of blocks to delete