[HDFS-17297] The NameNode should remove block from the BlocksMap if the block is marked as deleted.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.9
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: namanode
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Haiyang Hu
• Reporter: Haiyang Hu
• Created: 19/Dec/23 04:16
• Updated: 28/Jan/24 01:07
• Resolved: 28/Dec/23 13:14

Description

When call internalReleaseLease method:

boolean internalReleaseLease(
...
int minLocationsNum = 1;
if (lastBlock.isStriped()) {
  minLocationsNum = ((BlockInfoStriped) lastBlock).getRealDataBlockNum();
}

if (uc.getNumExpectedLocations() < minLocationsNum &&
    lastBlock.getNumBytes() == 0) {
  // There is no datanode reported to this block.
  // may be client have crashed before writing data to pipeline.
  // This blocks doesn't need any recovery.
  // We can remove this block and close the file.
  pendingFile.removeLastBlock(lastBlock);
  finalizeINodeFileUnderConstruction(src, pendingFile,
      iip.getLatestSnapshotId(), false); 
...
}

if the condition `uc.getNumExpectedLocations() < minLocationsNum && lastBlock.getNumBytes() == 0` is met during the execution of UNDER_RECOVERY logic, the block is removed from the block list in the inode file and marked as deleted.
However it is not removed from the BlocksMap, it may cause memory leak.

Therefore it is necessary to remove the block from the BlocksMap at this point as well.

Issue Links

• links to: GitHub Pull Request #6369