[HDFS-4183] Throttle block recovery

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.23.4, 2.0.2-alpha
• Component/s: namenode
• Target Version/s: 2.1.0-beta
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 13/Nov/12 02:13
• Updated: 30/Jan/14 21:36
• Resolved: 30/Jan/14 21:36

Description

When a large number of files are abandoned without closing, a storm of lease expiration follows in about an hour (lease hard limit). For the last block of each file, block recovery is initiated and when the datanode is done, it calls commitBlockSynchronization() is called against namenode. A burst of these calls can slow down namenode considerably. We need to throttle block recovery and/or speed up the rate at which commitBlockSynchronization() is served.

Issue Links

• is duplicated by: HDFS-5790 LeaseManager.findPath is very slow when many leases need recovery