[HDFS-12044] Mismatch between BlockManager#maxReplicationStreams and ErasureCodingWorker.stripedReconstructionPool pool size causes slow and bursty recovery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha3
• Fix Version/s: 3.0.0-beta1
• Component/s: erasure-coding
• Labels: hdfs-ec-3.0-must-do
• Target Version/s: 3.0.0-beta1
• Assignee: Lei (Eddy) Xu
• Reporter: Lei (Eddy) Xu
• Created: 27/Jun/17 00:37
• Updated: 28/Jul/17 18:15
• Resolved: 28/Jul/17 17:53

Description

ErasureCodingWorker#stripedReconstructionPool is with corePoolSize=2 and maxPoolSize=8 as default. And it rejects more tasks if the queue is full.

When BlockManager#maxReplicationStream is larger than ErasureCodingWorker#stripedReconstructionPool#corePoolSize/maxPoolSize, for example, maxReplicationStream=20 and corePoolSize=2 , maxPoolSize=8. Meanwhile, NN sends up to maxTransfer reconstruction tasks to DN for each heartbeat, and it is calculated in FSNamesystem:

final int maxTransfer = blockManager.getMaxReplicationStreams() - xmitsInProgress;

However, at any giving time, {ErasureCodingWorker#stripedReconstructionPool takes 2 xmitInProcess. So for each heartbeat in 3s, NN will send about 20-2 = 18 reconstruction tasks to the DN, and DN throw away most of them if there were 8 tasks in the queue already. So NN needs to take longer to re-consider these blocks were under-replicated to schedule new tasks.

Issue Links

• is related to: HDFS-12215 DataNode#transferBlock does not create its daemon in the xceiver thread group
• is related to: HDFS-12208 NN should consider DataNode#xmitInProgress when placing new block