Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-3122

Block recovery with closeFile flag true can race with blockReport. Due to this blocks are getting marked as corrupt.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Problem
    • Affects Version/s: 0.23.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: datanode, namenode
    • Labels:
      None

      Description

      Block Report can race with Block Recovery with closeFile flag true.

      Block report generated just before block recovery at DN side and due to N/W problems, block report got delayed to NN.
      After this, recovery success and generation stamp modifies to new one.
      And primary DN invokes the commitBlockSynchronization and block got updated in NN side. Also block got marked as complete, since the closeFile flag was true. Updated with new genstamp.

      Now blockReport started processing at NN side. This particular block from RBW (when it generated the BR at DN), and file was completed at NN side.
      Finally block will be marked as corrupt because of genstamp mismatch.

       case RWR:
            if (!storedBlock.isComplete()) {
              return null; // not corrupt
            } else if (storedBlock.getGenerationStamp() != iblk.getGenerationStamp()) {
              return new BlockToMarkCorrupt(storedBlock,
                  "reported " + reportedState + " replica with genstamp " +
                  iblk.getGenerationStamp() + " does not match COMPLETE block's " +
                  "genstamp in block map " + storedBlock.getGenerationStamp());
            } else { // COMPLETE block, same genstamp
      

        Attachments

        1. blockCorrupt.txt
          3 kB
          Uma Maheswara Rao G

          Activity

            People

            • Assignee:
              umamaheswararao Uma Maheswara Rao G
              Reporter:
              umamaheswararao Uma Maheswara Rao G
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: