[HDDS-9291] Deny block read requests when block is marked as deleted on Datanodes

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Siyao Meng
• Created: 15/Sep/23 01:39
• Updated: 15/Sep/23 01:44

Description

Inspired by HDDS-9146 – a sneaky data loss issue where the symptom (data loss) doesn't appear until the container is closed, we might want to add a safety on DNs so that as soon as a block is marked as deleted on a DN, the DN should refuse to serve the block right away, and throw an exception back to the client (stating the block is marked deleted on this Ozone Datanode).

This will help us catch data loss issues similar to HDDS-9146 much sooner in UTs, integration tests, acceptance tests and even in production (where we could technically manually rescue the blocks on DNs before the container is closed and actually block deletion happened).

cc sumitagrawl erose swagle ghuangups szetszwo

Issue Links

• is related to: HDDS-9146 Potential data loss with HSync due to deletedTable entry having the same block as keyTable entry's