[HDFS-4075] Reduce recommissioning overhead

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.4, 2.0.2-alpha
• Fix Version/s: 2.0.3-alpha, 0.23.5
• Component/s: namenode
• Target Version/s: 2.0.3-alpha, 0.23.5
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 18/Oct/12 19:31
• Updated: 03/Sep/14 23:19
• Resolved: 06/Nov/12 20:01

Description

When datanodes are recommissioned,

{BlockManager#processOverReplicatedBlocksOnReCommission()}

is called for each rejoined node and excess blocks are added to the invalidate list. The problem is this is done while the namesystem write lock is held.

Issue Links

• is related to: HDFS-4080 Add a separate logger for block state change logs to enable turning off those logs