[HDFS-12092] VolumeScanner exits when block metadata file is corrupted on datanode.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.0
• Component/s: datanode, hdfs
• Assignee: Unassigned
• Reporter: Ashwin Ramesh
• Created: 05/Jul/17 22:46
• Updated: 05/Jul/17 22:53

Description

Restarted a datanode, corrupted the metafile for blk_1073741825 with something like echo 'zzzzzzzz' > blk_1073741825_1001.meta, and datanode logs reveal that the VolumeScanner exits due to an illegal argument exception. Here is the relevant trace:
------------------------------------------------------------------------------------------------------

2017-07-05 22:03:41,878 [VolumeScannerThread()] DEBUG datanode.VolumeScanner: start scanning block BP-955735389-###########-1494002319684:blk_1073741825_1001
2017-07-05 22:03:41,879 [VolumeScannerThread()] ERROR datanode.VolumeScanner: VolumeScanner() exiting because of exception 
java.lang.IllegalArgumentException: id=122 out of range [0, 5)
        at org.apache.hadoop.util.DataChecksum$Type.valueOf(DataChecksum.java:67)
        at org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:123)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:178)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:142)
        at org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:156)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.loadLastPartialChunkChecksum(FsVolumeImpl.java:1022)
        at org.apache.hadoop.hdfs.server.datanode.FinalizedReplica.getLastChecksumAndDataLen(FinalizedReplica.java:104)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:259)
        at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.scanBlock(VolumeScanner.java:484)
        at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:614)
        at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:699)
2017-07-05 22:03:41,879 [VolumeScannerThread()] INFO datanode.VolumeScanner: VolumeScanner() exiting.