[HDDS-7759] Improve Ozone Replication Manager

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Assignee: Unassigned
• Reporter: Ethan Rose
• Created: 09/Jan/23 16:50
• Updated: 03/Nov/23 07:21
• Resolved: 11/Jul/23 15:10

Description

Parent Jira to capture tasks related to migrating from Ozone's legacy replication manager to the new refactored modular replication manager.

Issue Links

• relates to: HDDS-8699 Further Replication Manager Improvements
• relates to: HDDS-7776 Container replication in push model
• split from: HDDS-6462 Phase II : Erasure Coding Offline Recovery & Read/Write Improvements
• split from: HDDS-6548 HDDS Stability Fixes and Improvements

Sub-Tasks

• Refine SCM handling of unhealthy container replicas: Resolved
• EC: ReplicationManager - add move manager for container move: Resolved
• Improve Handling of Unhealthy Container Replicas in the new RM: Resolved
• Ratis OverReplicationHandler should exclude stale replicas: Resolved
• UNHEALTHY replicas will not contribute to sufficient replication in RatisContainerReplicaCount: Resolved
• Intermittent failure in TestReplicationManager#testUnderReplicationQueuePopulated: Resolved
• Handle Mismatched Replicas (OPEN or CLOSING) of QUASI-CLOSED containers in RM: Resolved
• Modify Ratis Replication Handling in the new RM: Resolved
• Handle Replication of Unhealthy Replicas in RM: Resolved
• Force close QUASI_CLOSED replicas only when the container is CLOSED in Legacy RM: Resolved
• Add configuration flag to enable LegacyReplicationManager for RATIS containers: Resolved
• Force Close QUASI_CLOSED replicas of CLOSED containers in RM: Resolved
• UnhealthyReplicationProcessor retries failure without delay: Resolved
• Synchronize on containerInfo in ReplicationManager and MoveManager: Resolved
• Let RatisMisReplicationHandler use the new RatisContainerReplicaCount constructor: Resolved
• Move pendingOps into ContainerStateManagerImpl to ensure consistent state: Resolved
• ReplicationManager: Count a container once for missing, under, mis or over replicated: Resolved
• Check container replication health before scheduling move in MoveManager: Resolved
• Replace Usages of LegacyReplicationManager.MoveResult with MoveManager.MoveResult: Resolved
• Improve synchronization around command queue updates in Node Manager: Resolved
• ECReconstructionCoordinatorTask.runTask should catch Exception: Resolved
• ReplicationManager: Introduce basic limits on ReplicateContainer commands: Resolved
• Clean up replication logs: Resolved
• Integrate ContainerBalancer with MoveManager: Resolved
• Replication Manager: Make all handlers send commands immediately instead of returning commands: Resolved
• Inject MoveManager into ContainerBalancer: Resolved
• Replicate commands can be sent to dead maintenance modes if the same index is being decommissioned: Resolved
• ReplicationManager: Add RatisMisReplicationHandler into rm.processUnderReplicatedContainer: Resolved
• ReplicationManager: Datanode commands should be sent to nodeManager directly: Resolved
• Make deadlines inside MoveManager for move commands configurable: Resolved
• ECUnderReplicationHandler should consider commands already sent when processing the container: Resolved
• ReplicationManager: Throttle delete container commands from over replication handlers: Resolved
• Let ReplicationManager decide the timeout for commands in Datanodes: Resolved
• Delay Starting ContainerBalancer after SCM failover: Resolved
• ReplicationManager: Basic Throttling of EC Reconstruction commands: Resolved
• ReplicationManager: Add nodes to exclude list if they are overloaded: Resolved
• Let ContainerBalancer consider EC containers for balancing: Resolved
• Fix the space usage comparator in ContainerBalancerSelectionCriteria: Resolved
• ReplicationManager: Fix getContainerReplicationHealth() so that it builds ContainerCheckRequest correctly: Resolved
• ReplicationManager: Create ContainerReplicaOp with correct target Datanode: Resolved
• ReplicationManager: Use RM exclude list when getting target nodes for reconstruction: Resolved
• ReplicationManager: MisReplicationHandler should throw an exception if partially successful: Resolved
• ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes: Resolved
• ContainerBalancer should move only CLOSED replicas: Resolved
• Consider seperating Ratis and EC MisReplication Handling: Resolved
• ReplicationManager: Allow partial EC reconstruction if insufficient nodes available: Resolved
• ReplicationManager: EC Mis and Under replication handler should handle overloaded exceptions: Resolved
• Disable LegacyReplicationManager by default: Resolved
• ReplicationManager: RatisUnderReplication handler should not sort sources by BCSID: Resolved
• Ratis under replication handling in a rack aware environment doesn't work: Resolved
• Ensure replication processors use a single queue for each iteration: Resolved
• ReplicationManager: Add configurable global replication limit: Resolved
• Investigate possible race conditions on ContainerInfo in ContainerBalancer: Resolved
• Adjust replication queue limits for decommissioning nodes: Resolved
• ReplicationManager: Clear ContainerReplicaPendingOps when RM goes to running state: Resolved
• Provide more info in assertions: Resolved
• Datanode decommissioning blocked due to non-empty replica of deleting container: Resolved
• Add config for factor of scaling up replication queue/threads in decommissioning nodes: Resolved
• ReplicationManager should handle CLOSING containers that are empty: Resolved
• ReplicationManager: Pass used and excluded node separately for Under and Mis-Replication: Resolved
• Underreplication not fixed if all replicas start decommissioning: Resolved
• Thread pool size needs to be decreased in different order in ReplicationSupervisor: Resolved
• ReplicationManager: Change default command timeout to 10 minutes: Resolved
• Ratis underreplication due to maintenance is not deprioritised: Resolved
• EC: ReplicationManager - consider deprecating maintenance.replica.minimum: Resolved
• ReplicationManager: Use EC config scheme to adjust the weighting of reconstruction tasks: Resolved
• ReplicationManager: Unhealthy containers could block EC recovery in small clusters: Resolved
• Incorrect expectedNodes passed to InsufficientNodesException: Resolved
• Fix expectation in testUnderRepSentToOverRepHandlerIfNoNewNodes: Resolved
• Add metrics to ReplicationSupervisor for task count and max stream: Resolved
• Delete empty containers that are stuck in CLOSING state: Resolved
• Clean up replication code: Resolved
• ReplicationManager: Log overloaded commands at debug rather than info level: Resolved
• ReplicationManager: Add metric to count how often replication is throttled: Resolved
• Prepare for dynamic config in ReplicationManager: Resolved
• ReplicationManager: Fix metrics to work with new RM: Resolved
• ReplicationManager: Add metrics for partial replication / reconstruction and cluster limit: Resolved
• Orphan blocks can leave empty container stuck deleting forever: Resolved
• Replication limit should not be less than reconstruction weight: Resolved
• Add metrics to Container Balancer: Resolved
• NPE in SCMCommonPlacementPolicy#validateContainerPlacement: Resolved