[HDDS-8447] Datanodes should not process container deletes for failed volumes

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Ozone Datanode
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Sadanand Shenoy
• Reporter: Arun Sarin
• Created: 18/Apr/23 17:40
• Updated: 28/Jun/23 06:20
• Resolved: 28/Jun/23 06:20

Description

Before HDDS-6449, if there was an error deleting containers from datanodes, the operation would be abandoned, leaving behind container pieces which could be problematic on restart. After that fix, deletion is more robust and moves the container to a volume specific tmp directory before deletion. This requires the volume to be present and healthy otherwise it will fail and the DN will be stuck in a container delete retry loop. In this jira, we will:
1. Check if the volume is in the failed volume set before deleting a container.
2. Trigger an on-demand volume scan if there is an IO error deleting the container.

Issue Links

• is caused by: HDDS-6449 Failed container delete can leave artifacts on disk
• relates to: HDDS-8448 Send FCR to SCM on volume failure
• split from: HDDS-8441 [EC] Unhealthy replica not getting replaced by healthy replica when hdds.datanode.dir path is deleted
• links to: GitHub Pull Request #4901