Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10710

In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved with NN lock protection

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In BlockManager#rescanPostponedMisreplicatedBlocks(), start and end block counts should be get with the protect with lock. Or, log records like "-1 blocks are removed" which indicate minus blocks are removed could be generated.

      For example, following scenario:
      1. thread1 run long startPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount(); currently startPostponedMisReplicatedBlocksCount get the value 20.
      2. before thread1 run namesystem.writeLock(); , thread2 increment postponedMisreplicatedBlocksCount by 1, so postponedMisreplicatedBlocksCount is 21 now.
      3. thread1 end the iteration, but no postponed block is removed, so after run long endPostponedMisReplicatedBlocksCount = getPostponedMisreplicatedBlocksCount();, endPostponedMisReplicatedBlocksCount get the value of 21.
      4. thread 1 generate the log:

            LOG.info("Rescan of postponedMisreplicatedBlocks completed in " +
                (Time.monotonicNow() - startTimeRescanPostponedMisReplicatedBlocks) +
                " msecs. " + endPostponedMisReplicatedBlocksCount +
                " blocks are left. " + (startPostponedMisReplicatedBlocksCount -
                endPostponedMisReplicatedBlocksCount) + " blocks are removed.");
      

      Then, we'll get the log record like "-1 blocks are removed."

        Attachments

          Activity

            People

            • Assignee:
              demongaorui Rui Gao
              Reporter:
              demongaorui Rui Gao
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: