[HDFS-8770] ReplicationMonitor thread received Runtime exception: NullPointerException when BlockManager.chooseExcessReplicates

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.6.0, 2.7.0
• Component/s: namenode
• Assignee: chunde
• Reporter: chunde
• Due: 08/Jul/15
• Created: 14/Jul/15 07:45
• Updated: 16/Nov/15 17:59
• Resolved: 16/Nov/15 12:14

Description

Namenode shutdown when ReplicationMonitor thread received Runtime exception:

2015-07-08 16:43:55,167 ERROR org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: ReplicationMonitor thread received Runtime exception.
java.lang.NullPointerException
at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy.adjustSetsWithChosenReplica(BlockPlacementPolicy.java:189)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseExcessReplicates(BlockManager.java:2911)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processOverReplicatedBlock(BlockManager.java:2849)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processMisReplicatedBlock(BlockManager.java:2780)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.rescanPostponedMisreplicatedBlocks(BlockManager.java:1931)
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$ReplicationMonitor.run(BlockManager.java:3628)
at java.lang.Thread.run(Thread.java:744)

We use hadoop-2.6.0 configured with heterogeneous storages and setStoragePolicy some path One_SSD. When a block has excess replicated like 2 SSD replica on different rack(exactlyOne set) and 2 Disk on same rack(moreThanOne set), BlockPlacementPolicyDefault.chooseReplicaToDelete return null because only moreThanOne set be chosen to find SSD replica

Issue Links

• duplicates: HDFS-9313 Possible NullPointerException in BlockManager if no excess replica can be chosen
• duplicates: HDFS-9314 Improve BlockPlacementPolicyDefault's picking of excess replicas