[HDFS-16180] FsVolumeImpl.nextBlock should consider that the block meta file has been deleted.

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.0, 3.4.0
• Fix Version/s: 3.4.0
• Component/s: datanode
• Labels: pull-request-available
• Assignee: Max Xie
• Reporter: Max Xie
• Created: 19/Aug/21 09:44
• Updated: 24/Aug/21 03:16
• Resolved: 24/Aug/21 03:16

Description

In my cluster,  we found that when VolumeScanner run, sometime dn will throw some error log below

```
 
2021-08-19 08:00:11,549 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-1020175758-nnip-1597745872895 blk_1142977964_69237147 URI file:/disk1/dfs/data/current/BP-1020175758- nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964
2021-08-19 08:00:48,368 ERROR org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl: nextBlock(DS-060c8e4c-1ef6-49f5-91ef-91957356891a, BP-1020175758- nnip-1597745872895): I/O error
java.io.IOException: Meta file not found, blockFile=/disk1/dfs/data/current/BP-1020175758- nnip-1597745872895/current/finalized/subdir0/subdir21/blk_1142977964
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetUtil.findMetaFile(FsDatasetUtil.java:101)
at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl.nextBlock(FsVolumeImpl.java:809)
at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:528)
at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:628)
2021-08-19 08:00:48,368 WARN org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/disk1/dfs/data, DS-060c8e4c-1ef6-49f5-91ef-91957356891a): nextBlock error on org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl$BlockIteratorImpl@7febc6b4
```

When VolumeScanner scan block  blk_1142977964,  it has been deleted by datanode,  scanner can not find the meta file of blk_1142977964, so it throw these error log.

 

Maybe we should handle FileNotFoundException during nextblock to reduce error log and nextblock retry times.

 

Issue Links

• links to: GitHub Pull Request #3315