[HDDS-10239] Storage Container Reconciliation

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Ozone Datanode, SCM
• Labels: pull-request-available
• Assignee: Ethan Rose
• Reporter: Ethan Rose
• Created: 30/Jan/24 01:37
• Updated: 17/Oct/24 15:20

Description

Ideally, a healthy Ozone cluster would contain only open and closed containers. However, container replicas commonly end up with a mix of states including quasi-closed and unhealthy that the current system is not able to resolve to cleanly closed replicas. The cause of these states is often bugs or broad failure handling on the write path. While we should fix these causes, they raise the problem that Ozone is not able to reconcile these mismatched container states on its own, regardless of their cause. This has lead to significant complexity in the replication manager for how to handle cases where only quasi-closed and unhealthy replicas are available, especially in the case of decommissioning.

Even when all replicas are closed, the system assumes that these closed container replicas are equal with no way to verify this. Checksumming is done for individual chunks within each container, but if two container replicas somehow end up with chunks that differ in length or content despite being marked closed with local checksums matching, the system has no way to detect or resolve this anomaly.

This Jira proposes a container reconciliation protocol to solve these problems. After implementing the proposal:
1. It should be possible for a cluster to progress to a state where it has only properly replicated closed and open containers.
2. We can verify the equality and integrity of all closed containers.

The design doc is linked here as a markdown pull request for inline comments.

Issue Links

• is duplicated by: HDDS-9280 Quasi-closed container with unhealthy replicas may remain under-replicated in 4 node cluster
• relates to: HDDS-10931 Schedule on demand scan of containers after import
• relates to: HDDS-11593 Improve container scanner metrics
• relates to: HDDS-7094 Enable Datanode side CRC checks by default
• links to: GitHub Pull Request #6121

Sub-Tasks

• Add GitHub actions labeler for the reconciliation feature branch: Resolved
• Datanode reports Merkel Tree container summary to SCM during heartbeats: Resolved
• SCM and Datanode communication for reconciliation: Resolved
• Container Scanner should still scan unhealthy containers: Resolved
• Implement a basic Merkle Tree Manager: Resolved
• Implement framework for capturing Merkle Tree Metrics: Resolved
• Block deletion should update container merkle tree: Resolved
• Add a Datanode API to supply a merkle tree for a given container: Resolved
• Reconcile commands should be handled by datanode ReplicationSupervisor: Resolved
• Datanodes should generate initial container merkle tree during container close: Resolved
• Handle corrupted merkle tree files: Resolved
• Container scanner should keep scanning after non-fatal errors: Resolved
• Allow datanodes to do chunk level modifications to closed containers: Resolved
• Implement container comparison and repair logic within datanodes: Patch Available
• Add new tests for container scanner detecting multiple errors in one container: Patch Available
• Make container scanner generate merkle trees during the scan: Open
• Coordinate container reconciliation with container deletion and replication: Open
• Allow reconciliation and scanner to move replicas out of the UNHEALTHY state: Open
• Handle backwards compatibility for containers created before reconciliation: Open
• Consider allowing reconciliation when not all replicas have reached closed state: Open
• SCMExceptions resulting from admin CLI commands are treated as retriable: Open
• Restrict reconciliation requests by datanode status: Open
• Extend container repair capabilities to the block level: Open
• Combine datanode clients for reconciliation and EC reconstruction: Open
• Basic SCM co-ordination: Open
• Optimize checksum calculations in container merkle tree: Open
• Add metrics specific to reconciliation tasks: Open
• Use zero-copy for readMerkleTree API: Open