Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0
-
Reviewed
Description
We encounterd several missing-block problem in our production cluster which hdfs running on AWS EC2 + EBS.
The root cause:
- the block remains only 1 replication left and hasn't been reconstruction
- DN checks block file existing when BlockSender construction
- the EBS checking failed and throw FileNotFoundException (EBS may be in fault condition)
- DN invalidateBlock and schedule block async deletion
- EBS already back to normal when DN do delete block
- the block file be delete permanently and can't be recovered
Attachments
Issue Links
- links to