[HDFS-6425] Large postponedMisreplicatedBlocks has impact on blockReport latency

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.7.0
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 17/May/14 05:39
• Updated: 10/Mar/16 23:56
• Resolved: 16/Dec/14 16:33

Description

Sometimes we have large number of over replicates when NN fails over. When the new active NN took over, over replicated blocks will be put to postponedMisreplicatedBlocks until all DNs for that block aren't stale anymore.

We have a case where NNs flip flop. Before postponedMisreplicatedBlocks became empty, NN fail over again and again. So postponedMisreplicatedBlocks just kept increasing until the cluster is stable.

In addition, large postponedMisreplicatedBlocks could make rescanPostponedMisreplicatedBlocks slow. rescanPostponedMisreplicatedBlocks takes write lock. So it could slow down the block report processing.

Issue Links

• is duplicated by: HDFS-7421 Move processing of postponed over-replicated blocks to a background task
• is related to: HDFS-9434 Recommission a datanode with 500k blocks may pause NN for 30 seconds