Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
See HADOOP-2576 and HADOOP-774 for more discussion.
Namenode throttles the number of blocks it asks Datanode to delete. It does this because it "knows" that Datanode deletes these blocks in the same thread that heartbeats and does not want that thread to block for long. Managing this is more memory and more code at Namenode.
I think namenode should just ask Datanode to delete the blocks and Datanode can decide how it deletes them. It would be datanode's responsibility to properly delete the blocks however it sees fit. For e.g. it could delete them in separate thread and not let heartbeats affected by this.