Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14631

The DirectoryScanner doesn't fix the wrongly placed replica.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.10.0, 3.3.0, 3.2.1, 2.9.3, 3.1.3
    • Component/s: None
    • Labels:
      None

      Description

      When DirectoryScanner scans block files, if the block refers to the block file does not exist the DirectoryScanner will update the block based on the replica file found on the disk. See FsDatasetImpl#checkAndUpdate.

       

      /*
      * Block exists in volumeMap and the block file exists on the disk
      */
      // Compare block files
      if (memBlockInfo.blockDataExists()) {
        ...
      } else {
        // Block refers to a block file that does not exist.
        // Update the block with the file found on the disk. Since the block
        // file and metadata file are found as a pair on the disk, update
        // the block based on the metadata file found on the disk
        LOG.warn("Block file in replica "
            + memBlockInfo.getBlockURI()
            + " does not exist. Updating it to the file found during scan "
            + diskFile.getAbsolutePath());
        memBlockInfo.updateWithReplica(
        StorageLocation.parse(diskFile.toString()));
      
        LOG.warn("Updating generation stamp for block " + blockId
            + " from " + memBlockInfo.getGenerationStamp() + " to " + diskGS);
        memBlockInfo.setGenerationStamp(diskGS);
      }
      

      But the DirectoryScanner doesn't really fix it because in LocalReplica#parseBaseDir() the 'subdir' are ignored.

       

       

        Attachments

        1. HDFS-14631-branch-2.9.001.patch
          6 kB
          Jinglun
        2. HDFS-14631.004.patch
          6 kB
          Jinglun
        3. HDFS-14631.003.patch
          6 kB
          Jinglun
        4. HDFS-14631.002.patch
          6 kB
          Jinglun
        5. HDFS-14631.001.patch
          6 kB
          Jinglun

          Activity

            People

            • Assignee:
              LiJinglun Jinglun
              Reporter:
              LiJinglun Jinglun
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: