[HDFS-14631] The DirectoryScanner doesn't fix the wrongly placed replica.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 3.3.0, 3.2.1, 2.9.3, 3.1.3
• Assignee: Jinglun
• Reporter: Jinglun
• Created: 04/Jul/19 10:55
• Updated: 07/Aug/19 20:48
• Resolved: 07/Aug/19 20:48

Description

When DirectoryScanner scans block files, if the block refers to the block file does not exist the DirectoryScanner will update the block based on the replica file found on the disk. See FsDatasetImpl#checkAndUpdate.

 

/*
* Block exists in volumeMap and the block file exists on the disk
*/
// Compare block files
if (memBlockInfo.blockDataExists()) {
  ...
} else {
  // Block refers to a block file that does not exist.
  // Update the block with the file found on the disk. Since the block
  // file and metadata file are found as a pair on the disk, update
  // the block based on the metadata file found on the disk
  LOG.warn("Block file in replica "
      + memBlockInfo.getBlockURI()
      + " does not exist. Updating it to the file found during scan "
      + diskFile.getAbsolutePath());
  memBlockInfo.updateWithReplica(
  StorageLocation.parse(diskFile.toString()));

  LOG.warn("Updating generation stamp for block " + blockId
      + " from " + memBlockInfo.getGenerationStamp() + " to " + diskGS);
  memBlockInfo.setGenerationStamp(diskGS);
}

But the DirectoryScanner doesn't really fix it because in LocalReplica#parseBaseDir() the 'subdir' are ignored.