[HDDS-9254] Legacy replication manager uses mismatched replicas as replication sources

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Ethan Rose
• Reporter: Ethan Rose
• Created: 07/Sep/23 20:15
• Updated: 31/Oct/23 14:06
• Resolved: 19/Sep/23 21:08

Description

Consider a case where SCM has a CLOSED container and the replica states are CLOSED, CLOSED, QUASI. In the pull replication model, RM will send all 3 of these replicas to the datanode to use as replication sources. The DN will do a random shuffle and pick one to replicate. If it chooses the QUASI-CLOSED replica, the next iteration of RM will see replicas CLOSED, CLOSED, QUASI, QUASI. RM will issue the same command since the CLOSED replicas are still under replicated, but now the odds of the DN's random shuffle choosing a quasi closed replica are increased. This process can repeat until the cluster is filled with a quasi-closed replica on each datanode. This can bring the cluster into the stuck state described in HDDS-8536.

Issue Links

• is caused by: HDDS-6447 Refine SCM handling of unhealthy container replicas
• relates to: HDDS-9591 Replication Manager could incorrectly use QUASI_CLOSED replicas as replication sources for CLOSED containers
• links to: GitHub Pull Request #5257