[HDFS-17150] EC: Fix the bug of failed lease recovery.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.9
• Fix Version/s: 3.4.0, 3.3.9
• Component/s: erasure-coding
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Shuyan Zhang Assign to me
• Reporter: Shuyan Zhang
• Created: 10/Aug/23 03:10
• Updated: 28/Jan/24 01:47
• Resolved: 15/Aug/23 11:55

Description

If the client crashes without writing the minimum number of internal blocks required by the EC policy, the lease recovery process for the corresponding unclosed file may continue to fail. Taking RS(6,3) policy as an example, the timeline is as follows:
1. The client writes some data to only 5 datanodes;
2. Client crashes;
3. NN fails over;
4. Now the result of `uc.getNumExpectedLocations()` completely depends on block report, and there are 5 datanodes reporting internal blocks;
5. When the lease expires hard limit, NN issues a block recovery command;
6. The datanode checks the command and finds that the number of internal blocks is insufficient, resulting in an error and recovery failure;

7. The lease expires hard limit again, and NN issues a block recovery command again, but the recovery fails again......

When the number of internal blocks written by the client is less than 6, the block group is actually unrecoverable. We should equate this situation to the case where the number of replicas is 0 when processing replica files, i.e., directly remove the last block group and close the file.

 

Issue Links

• links to: GitHub Pull Request #5937