[HDDS-1595] Handling IO Failures on the Datanode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Component/s: Ozone Datanode
• Target Version/s: 0.5.0
• Assignee: Supratim Deka
• Reporter: Supratim Deka
• Created: 27/May/19 12:30
• Updated: 04/Dec/23 02:59
• Resolved: 01/Jun/20 02:37

Description

This Jira covers all the changes required to handle IO Failures on the Datanode. Handling an IO failure on the Datanode involves detecting failures as they happen and propagating the failure to the appropriate component in the system - possibly the Client and/or the SCM based on the type of failure.

At a high-level, IO Failure handling has the following goals:
1. Prevent Inconsistencies and corruption - due to non-handling or mishandling of failures.
2. Prevent any data loss - timely detection of failure and propagate correct error back to the initiator instead of silently dropping the data while the client assumes the operation is committed.
3. Contain the disruption in the system - if a disk volume fails on a DN, operations to the other nodes and volumes should not be affected.

Details pertaining to design and changes required are covered in the attached pdf document.
A sequence diagram used to analyse the Datanode IO Path is also attached, in svg format.

Issue Links

• incorporates: RATIS-573 Handle Raft Log Append Failure
• is related to: HDDS-9821 XceiverServerRatis SyncTimeoutRetry is overridden
• is related to: RATIS-1947 TimeoutIOException in WriteLog might not release Pending Requests

Sub-Tasks

• writeData in ChunkUtils should not use AsynchronousFileChannel: Resolved
• Handle Ratis Append Failure in Container State Machine: Resolved
• Propagate failure in writeStateMachineData to Ratis: Resolved
• Handle Apply Transaction Failure in State Machine: Resolved
• Handle Failure to Update Ozone Container YAML: Resolved