[HDFS-16985] Fix data missing issue when delete local block file.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: datanode
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Chengwei Wang
• Reporter: Chengwei Wang
• Created: 18/Apr/23 02:27
• Updated: 28/Jan/24 07:07
• Resolved: 14/May/23 13:35

Description

We encounterd several missing-block problem in our production cluster which  hdfs  running on AWS EC2 + EBS.

The root cause：

1. the block remains only 1 replication left and hasn't been reconstruction
2. DN checks block file existing when BlockSender construction
3. the EBS checking failed and throw FileNotFoundException (EBS may be in fault condition)
4. DN invalidateBlock and schedule block  async deletion
5. EBS already back to normal when DN do delete block
6. the block file be delete permanently and can't be recovered

Issue Links

• links to: GitHub Pull Request #5564