[HDFS-17298] Fix NPE in DataNode.handleBadBlock and BlockSender

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.9
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: datanode
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Haiyang Hu
• Reporter: Haiyang Hu
• Created: 20/Dec/23 12:04
• Updated: 28/Jan/24 01:07
• Resolved: 26/Dec/23 00:44

Description

There are some NPE issues on the DataNode side of our online environment.

The detailed exception information is

2023-12-20 13:58:25,449 ERROR datanode.DataNode (DataXceiver.java:run(330)) [DataXceiver for client DFSClient_NONMAPREDUCE_xxx at /xxx:41452 [Sending block BP-xxx:blk_xxx]] - xxx:50010:DataXceiver error processing READ_BLOCK operation  src: /xxx:41452 dst: /xxx:50010
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:301)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:607)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:152)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:104)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:298)
        at java.lang.Thread.run(Thread.java:748)

NPE Code logic:

if (!fromScanner && blockScanner.isEnabled()) {
  // data.getVolume(block) is null
  blockScanner.markSuspectBlock(data.getVolume(block).getStorageID(),
      block);
} 

2023-12-20 13:52:18,844 ERROR datanode.DataNode (DataXceiver.java:run(330)) [DataXceiver for client /xxx:61052 [Copying block BP-xxx:blk_xxx]] - xxx:50010:DataXceiver error processing COPY_BLOCK operation  src: /xxx:61052 dst: /xxx:50010
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.datanode.DataNode.handleBadBlock(DataNode.java:4045)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.copyBlock(DataXceiver.java:1163)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opCopyBlock(Receiver.java:291)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:113)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:298)
        at java.lang.Thread.run(Thread.java:748)

NPE Code logic:

// Obtain a reference before reading data
volumeRef = datanode.data.getVolume(block).obtainReference(); //datanode.data.getVolume(block) is null  

We need to fix it.

Issue Links

• links to: GitHub Pull Request #6374