[HDFS-9190] VolumeScanner throwing NPE while scanning suspect block.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.7.0
• Component/s: datanode
• Target Version/s: 2.8.0, 2.7.2
• Assignee: Unassigned
• Reporter: Rushabh Shah
• Created: 02/Oct/15 17:17
• Updated: 24/Nov/15 03:40
• Resolved: 02/Oct/15 18:11

Description

Volume scanner NPEs while scanning suspect Block.
Following is the stack trace:

2015-10-02 06:45:30,333 [VolumeScannerThread(dataDir)] ERROR datanode.VolumeScanner: VolumeScanner(dataDir, DS-5fc4263e-7a5c-4463-9f82-842108c0ab3b) exiting because of exception 
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.runLoop(VolumeScanner.java:539)
        at org.apache.hadoop.hdfs.server.datanode.VolumeScanner.run(VolumeScanner.java:619)
2015-10-02 06:45:30,333 [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@7768ca5] WARN datanode.DataNode: DatanodeRegistration(sourceDN:1004, datanodeUuid=f554982f-7c45-4fd4-ad57-9d472a39729e, infoPort=1006, infoSecurePort=0, ipcPort=8020, storageInfo=lv=-56;cid=CID-ddc217ab-5203-48ef-9695-a348feb4dac2;nsid=1872110141;c=1443758672580):Failed to transfer BP-1749317823-<nnIp>-1443758669533:blk_1073742231_1407 to destDN:1004 got 
java.net.SocketException: Original Exception : java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:443)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:575)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:579)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:759)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:706)
        at org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:2124)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection reset by peer
        ... 9 more

It is NPEing at the code in file: VolumeScanner#runLoop

      long saveDelta = monotonicMs - curBlockIter.getLastSavedMs();

curBlockIter is not initialized.

Issue Links

• is broken by: HDFS-7430 Rewrite the BlockScanner to use O(1) memory and use multiple threads
• is duplicated by: HDFS-8850 VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks