[HDFS-9958] BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.2
• Fix Version/s: 2.8.0, 2.7.3, 3.0.0-alpha1
• Target Version/s: 2.7.3
• Hadoop Flags: Reviewed
• Assignee: Kuhu Shukla
• Reporter: Kuhu Shukla
• Created: 14/Mar/16 19:11
• Updated: 23/Sep/19 23:51
• Resolved: 28/Apr/16 21:51

Description

In a scenario where the corrupt replica is on a failed storage, before it is taken out of blocksMap, there is a race which causes the creation of LocatedBlock on a machines array element that is not populated.

Following is the root cause,

final int numCorruptNodes = countNodes(blk).corruptReplicas();

countNodes only looks at nodes with storage state as NORMAL, which in the case where corrupt replica is on failed storage will amount to numCorruptNodes being zero.

final int numNodes = blocksMap.numNodes(blk);

However, numNodes will count all nodes/storages irrespective of the state of the storage. Therefore numMachines will include such (failed) nodes. The assert would fail only if the system is enabled to catch Assertion errors, otherwise it goes ahead and tries to create LocatedBlock object for that is not put in the machines array.

Here is the stack trace:

java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.toDatanodeInfos(DatanodeStorageInfo.java:45)
	at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeStorageInfo.toDatanodeInfos(DatanodeStorageInfo.java:40)
	at org.apache.hadoop.hdfs.protocol.LocatedBlock.<init>(LocatedBlock.java:84)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:878)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:826)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:799)
	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:899)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1849)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:588)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

Issue Links

• is duplicated by: HDFS-10788 fsck NullPointerException when it encounters corrupt replicas
• is related to: HDFS-14778 BlockManager findAndMarkBlockAsCorrupt adds block to the map if the Storage state is failed
• relates to: HDFS-10316 revisit corrupt replicas count
• relates to: HDFS-10342 BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt