[HDFS-11827] NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.9.0, 3.0.0-alpha4

Environment

hadoop-version: 2.6.0-cdh5.4.4

• Hadoop Flags: Reviewed
• Assignee: xupeng
• Reporter: xupeng
• Created: 16/May/17 06:28
• Updated: 08/Aug/18 22:22
• Resolved: 17/May/17 02:53

Description

This issue was found by my colleague when changing log-level of BlockPlacementPolicy using "hadoop daemonlog" command.

The exception stack trace is below:

2017-05-16 13:36:51,053 ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception.
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:666)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRemoteRack(BlockPlacementPolicyDefault.java:607)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:375)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:446)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:446)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:240)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:119)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithMultiDC.chooseTargetInternal(BlockPlacementPolicyWithMultiDC.java:263)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyWithMultiDC.chooseTarget(BlockPlacementPolicyWithMultiDC.java:214)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.chooseTargets(BlockManager.java:3836)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationWork.access$200(BlockManager.java:3801)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1394)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1300)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3764)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3711)
        at java.lang.Thread.run(Thread.java:745)

After checking BlockPlacementPolicyDefault code of trunk, I found that BlockPlacementPolicyDefault class missed some NPE check code in chooseRandom() method.

Issue Links

• is related to: HDFS-12416 BlockPlacementPolicyDefault will cause NN shutdown if log level is changed
• relates to: HDFS-12177 NameNode exits due to setting BlockPlacementPolicy loglevel to Debug