Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14852

Removing from LowRedundancyBlocks does not remove the block from all queues

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0, 3.0.3, 3.1.2, 3.3.0
    • 3.2.2, 3.3.1, 3.4.0
    • namenode
    • None

    Description

      LowRedundancyBlocks.java

      // Some comments here
          if(priLevel >= 0 && priLevel < LEVEL
              && priorityQueues.get(priLevel).remove(block)) {
            NameNode.blockStateChangeLog.debug(
                "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
                    + " from priority queue {}",
                block, priLevel);
            decrementBlockStat(block, priLevel, oldExpectedReplicas);
            return true;
          } else {
            // Try to remove the block from all queues if the block was
            // not found in the queue for the given priority level.
            for (int i = 0; i < LEVEL; i++) {
              if (i != priLevel && priorityQueues.get(i).remove(block)) {
                NameNode.blockStateChangeLog.debug(
                    "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
                        " {} from priority queue {}", block, i);
                decrementBlockStat(block, i, oldExpectedReplicas);
                return true;
              }
            }
          }
          return false;
        }
      

      Source code is above, the comments as follow

      // Try to remove the block from all queues if the block was
      // not found in the queue for the given priority level.

      The function "remove" does NOT remove the block from all queues.

      Function add from LowRedundancyBlocks.java is used on some places and maybe one block in two or more queues.
      We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is related to this.

      Upload initial patch

      Attachments

        1. CorruptBlocksMismatch.png
          166 kB
          Hui Fei
        2. HDFS-14852.001.patch
          4 kB
          Hui Fei
        3. HDFS-14852.002.patch
          4 kB
          Hui Fei
        4. HDFS-14852.003.patch
          4 kB
          Hui Fei
        5. HDFS-14852.004.patch
          3 kB
          Hui Fei
        6. HDFS-14852.005.patch
          3 kB
          Hui Fei
        7. HDFS-14852.006.patch
          4 kB
          Hui Fei
        8. HDFS-14852.007.patch
          3 kB
          Hui Fei
        9. screenshot-1.png
          192 kB
          Hui Fei

        Activity

          People

            ferhui Hui Fei
            ferhui Hui Fei
            Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: