[HDFS-14446] No enough internal blocks causes NN infinitely loop on releasing lease

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.0
• Component/s: datanode, erasure-coding, namenode
• Assignee: Unassigned
• Reporter: HuangTao
• Created: 21/Apr/19 04:51
• Updated: 21/Apr/19 04:53

Description

[2019-04-20T18:59:16.564+08:00] [INFO] [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@44e74aa0] : BlockRecoveryWorker: NameNode at /XXX.YYY.134.11:8021 calls recoverBlock(BP-1516815134-XXX.YYY.134.11-1555507434409:blk_-9223372036854775424_1037, targets=[DatanodeInfoWithStorage[XXX.YYY.134.135:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.202:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.196:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.197:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.138:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.199:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.235:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.134:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.35:50010,null,null]], newGenerationStamp=1038, newBlock=null, isStriped=true) [2019-04-20T18:59:17.298+08:00] [INFO] [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@44e74aa0] : initReplicaRecovery: blk_-9223372036854775417_1037, recoveryId=1038, replica=null [2019-04-20T18:59:17.398+08:00] [WARN] [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@44e74aa0] : recoverBlocks FAILED: RecoveringStripedBlock{BP-1516815134-XXX.YYY.134.11-1555507434409:blk_-9223372036854775424_1037; getBlockSize()=0; corrupt=false; offset=-1; locs=[DatanodeInfoWithStorage[XXX.YYY.134.135:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.202:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.196:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.197:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.138:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.199:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.235:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.134:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.35:50010,null,null]]} java.io.IOException: BP-1516815134-XXX.YYY.134.11-1555507434409:blk_-9223372036854775424_1037 has no enough internal blocks, unable to start recovery. Locations=[DatanodeInfoWithStorage[XXX.YYY.134.135:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.202:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.196:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.197:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.138:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.199:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.235:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.134:50010,null,null], DatanodeInfoWithStorage[XXX.YYY.134.35:50010,null,null]] at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskStriped.checkLocations(BlockRecoveryWorker.java:532) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskStriped.recover(BlockRecoveryWorker.java:437) at org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:602) at java.lang.Thread.run(Thread.java:745)

NN try to recovery UnderConstruction erasure-coding file, but the last block has no RBW files wrote on DN. The DN can't find internal blocks and won't tell NN to delete the last block, which will cause NN infinitely loop on releasing the file's lease.