[HDFS-12177] NameNode exits due to setting BlockPlacementPolicy loglevel to Debug

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.7.4, 2.8.2
• Component/s: block placement
• Hadoop Flags: Reviewed
• Assignee: Jiandan Yang
• Reporter: Jiandan Yang
• Created: 21/Jul/17 02:50
• Updated: 08/Aug/18 22:22
• Resolved: 21/Jul/17 13:27

Description

NameNode exits because the ReplicationMonitor thread internally throws NPE.
The reason for throwing NPE is that the builder field is not initialized whe do log.
Solution: before appending it should determine whether the builder is null

if (LOG.isDebugEnabled()) {
  builder = debugLoggingBuilder.get();
  builder.setLength(0);
  builder.append("[");
}
some other codes ...
if (LOG.isDebugEnabled()) {
  builder.append("\nNode ").append(NodeBase.getPath(chosenNode))
      .append(" [");
}
some other codes ...
if (LOG.isDebugEnabled()) {
  builder.append("\n]");
}

NN exception log is :

java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:722)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:689)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseFromNextRack(BlockPlacementPolicyDefault.java:640)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:608)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:483)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:390)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:419)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:266)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:119)
        at org.apache.hadoop.hdfs.server.blockmanagement.ReplicationWork.chooseTargets(ReplicationWork.java:55)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWorkForBlocks(BlockManager.java:1532)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeReplicationWork(BlockManager.java:1491)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.computeDatanodeWork(BlockManager.java:3768)
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3720)
        at java.lang.Thread.run(Thread.java:834)

Issue Links

• is related to: HDFS-11827 NPE is thrown when log level changed in BlockPlacementPolicyDefault#chooseRandom() method
• is related to: HDFS-12416 BlockPlacementPolicyDefault will cause NN shutdown if log level is changed