[HDFS-10710] In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved with NN lock protection

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: namenode
• Hadoop Flags: Reviewed
• Assignee: Rui Gao
• Reporter: Rui Gao
• Created: 01/Aug/16 09:36
• Updated: 30/Aug/16 01:14
• Resolved: 03/Aug/16 19:46

Description

In BlockManager#rescanPostponedMisreplicatedBlocks(), start and end block counts should be get with the protect with lock. Or, log records like "-1 blocks are removed" which indicate minus blocks are removed could be generated.

For example, following scenario:
1. thread1 run long startPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount(); currently startPostponedMisReplicatedBlocksCount get the value 20.
2. before thread1 run namesystem.writeLock(); , thread2 increment postponedMisreplicatedBlocksCount by 1, so postponedMisreplicatedBlocksCount is 21 now.
3. thread1 end the iteration, but no postponed block is removed, so after run long endPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount();, endPostponedMisReplicatedBlocksCount get the value of 21.
4. thread 1 generate the log:

      LOG.info("Rescan of postponedMisreplicatedBlocks completed in " +
          (Time.monotonicNow() - startTimeRescanPostponedMisReplicatedBlocks) +
          " msecs. " + endPostponedMisReplicatedBlocksCount +
          " blocks are left. " + (startPostponedMisReplicatedBlocksCount -
          endPostponedMisReplicatedBlocksCount) + " blocks are removed.");

Then, we'll get the log record like "-1 blocks are removed."