[HDDS-9595] Investigate QUASI_CLOSED containers with only one UNHEALTHY and empty replica

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SCM
• Target Version/s: 1.5.0
• Assignee: Unassigned
• Reporter: Siddhant Sangwan
• Created: 31/Oct/23 14:49
• Updated: 16/Dec/23 23:43

Description

We've seen some QUASI_CLOSED containers that have only 1 replica, and that 1 replica is UNHEALTHY and also empty. We need to investigate how the system ends up having such containers, and what to do about them. The solution will likely depend on whether the container is known by SCM to have zero keys. In general, this ties up into the larger problem of the SCM not knowing which containers that appear empty are actually not empty but have missing data, because it doesn't know if there are keys mapped to this container.

Currently, this is our logic for calling a container empty:

    return container.getState() == HddsProtos.LifeCycleState.CLOSED &&
        !replicas.isEmpty() &&
        replicas.stream().allMatch(
            r -> r.getState() == ContainerReplicaProto.State.CLOSED &&
                r.isEmpty());

Issue Links

• requires: HDDS-9596 Determine the criteria for calling a container empty in the SCM