[HDFS-15725] Lease Recovery never completes for a committed block which the DNs never finalize

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.3.1, 3.4.0, 2.10.2, 3.2.3
• Component/s: namenode
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 10/Dec/20 17:57
• Updated: 27/Jan/24 02:56
• Resolved: 14/Dec/20 12:49

Description

It a very rare condition, the HDFS client process can get killed right at the time it is completing a block / file.

The client sends the "complete" call to the namenode, moving the block into a committed state, but it dies before it can send the final packet to the Datanodes telling them to finalize the block.

This means the blocks are stuck on the datanodes in RBW state and nothing will ever tell them to move out of that state.

The namenode / lease manager will retry forever to close the file, but it will always complain it is waiting for blocks to reach minimal replication.

I have a simple test and patch to fix this, but I think it warrants some discussion on whether this is the correct thing to do, or if I need to put the fix behind a config switch.

My idea, is that if lease recovery occurs, and the block is still waiting on "minimal replication", just put the file back to UNDER_CONSTRUCTION so that on the next lease recovery attempt, BLOCK RECOVERY will happen, close the file and move the replicas to FINALIZED.

Issue Links

• relates to: HDFS-15726 Client should only complete a file if the last block is finalized
• relates to: HDFS-14498 LeaseManager can loop forever on the file for which create has failed