[HDFS-6833] DirectoryScanner should not register a deleting block with memory of DataNode

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5.0, 2.5.1, 3.0.0-alpha1
• Fix Version/s: 2.7.0
• Component/s: datanode
• Target Version/s: 2.6.1
• Hadoop Flags: Reviewed
• Assignee: Shinichi Yamashita
• Reporter: Shinichi Yamashita
• Created: 07/Aug/14 11:33
• Updated: 12/May/16 18:12
• Resolved: 12/Mar/15 18:32

Description

When a block is deleted in DataNode, the following messages are usually output.

2014-08-07 17:53:11,606 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825 for deletion
2014-08-07 17:53:11,617 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825

However, DirectoryScanner may be executed when DataNode deletes the block in the current implementation. And the following messsages are output.

2014-08-07 17:53:30,519 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825 for deletion
2014-08-07 17:53:31,426 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-1887080305-172.28.0.101-1407398838872 Total blocks: 1, missing metadata files:0, missing block files:0, missing blocks in memory:1, mismatched blocks:0
2014-08-07 17:53:31,426 WARN org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added missing block to memory FinalizedReplica, blk_1073741825_1001, FINALIZED
  getNumBytes()     = 21230663
  getBytesOnDisk()  = 21230663
  getVisibleLength()= 21230663
  getVolume()       = /hadoop/data1/dfs/data/current
  getBlockFile()    = /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825
  unlinked          =false
2014-08-07 17:53:31,531 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1887080305-172.28.0.101-1407398838872 blk_1073741825_1001 file /hadoop/data1/dfs/data/current/BP-1887080305-172.28.0.101-1407398838872/current/finalized/subdir0/subdir0/blk_1073741825

Deleting block information is registered in DataNode's memory.
And when DataNode sends a block report, NameNode receives wrong block information.

For example, when we execute recommission or change the number of replication, NameNode may delete the right block as "ExcessReplicate" by this problem.
And "Under-Replicated Blocks" and "Missing Blocks" occur.

When DataNode run DirectoryScanner, DataNode should not register a deleting block.