[HDDS-8536] ReplicationManager: Unhealthy replicas could block Ratis containers being recovered

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SCM
• Labels: pull-request-available
• Target Version/s: 1.4.0
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 04/May/23 15:20
• Updated: 21/Jan/24 15:48
• Resolved: 12/Sep/23 05:49

Description

In a similar way to HDDS-8535, if the cluster is small, say 4 nodes and a Ratis container has 2 unhealthy containers, RM will currently recover one new replia, leaving all 4 nodes used with 2 healthy and 2 unhealthy. As unhealthy containers are only removed after all over and under replication has been resolved, the container will remain stuck like this.

To avoid this, if there are insufficient spare nodes and also some unhealthy containers, then the under replication handler may need to call into the unhealthy handler to remove some of the unhealthy replicas to allow progress to be made.

Issue Links

• relates to: HDDS-9257 LegacyReplicationManager: Unhealthy replicas could block under replication handling
• links to: GitHub Pull Request #5255