[HDFS-11499] Decommissioning stuck because of failing recovery

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.1, 2.7.2, 2.7.3, 3.0.0-alpha2
• Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
• Component/s: hdfs, namenode
• Labels: blockmanagement decommission recovery
• Hadoop Flags: Reviewed

Release Note

Allow a block to complete if the number of replicas on live nodes, decommissioning nodes and nodes in maintenance mode satisfies minimum replication factor.
The fix prevents block recovery failure if replica of last block is being decommissioned. Vice versa, the decommissioning will be stuck, waiting for the last block to be completed. In addition, file close() operation will not fail due to last block being decommissioned.

• Assignee: Lukas Majercak
• Reporter: Lukas Majercak
• Created: 05/Mar/17 20:01
• Updated: 02/Oct/19 17:13
• Resolved: 13/Mar/17 20:49

Description

Block recovery will fail to finalize the file if the locations of the last, incomplete block are being decommissioned. Vice versa, the decommissioning will be stuck, waiting for the last block to be completed.

org.apache.hadoop.ipc.RemoteException(java.lang.IllegalStateException): Failed to finalize INodeFile testRecoveryFile since blocks[255] is non-complete, where blocks=[blk_1073741825_1001, blk_1073741826_1002...

The fix is to count replicas on decommissioning nodes when completing last block in BlockManager.commitOrCompleteLastBlock, as we know that the DecommissionManager will not decommission a node that has UC blocks.

Issue Links

• is duplicated by: HDFS-11486 Client close() should not fail fast if the last block is being decommissioned
• is duplicated by: HDFS-11852 Under-repicated block never completes because of failure in commitBlockSynchronization()
• is related to: HDFS-9392 Admins support for maintenance state
• is related to: HDFS-11486 Client close() should not fail fast if the last block is being decommissioned
• is related to: HDFS-14429 Block remain in COMMITTED but not COMPLETE caused by Decommission
• relates to: HDFS-12747 Lease monitor may infinitely loop on the same lease