[HDDS-8336] ReplicationManager: RatisUnderReplicationHandler should partially recover the container if not enough nodes

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: SCM
• Labels: pull-request-available
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 30/Mar/23 16:51
• Updated: 12/Apr/23 19:11
• Resolved: 12/Apr/23 19:11

Description

In RatisUnderReplicationHandler, if the container is under-replicated by more than 1 replica, then we need to select 2 or more new copies. In the code getTargets():

  private List<DatanodeDetails> getTargets(
      RatisContainerReplicaCount replicaCount,
      List<ContainerReplicaOp> pendingOps) throws IOException {
    // DNs that already have replicas cannot be targets and should be excluded
    final List<DatanodeDetails> excludeList =
        replicaCount.getReplicas().stream()
            .map(ContainerReplica::getDatanodeDetails)
            .collect(Collectors.toList());

    // DNs that are already waiting to receive replicas cannot be targets
    final List<DatanodeDetails> pendingReplication =
        pendingOps.stream()
            .filter(containerReplicaOp -> containerReplicaOp.getOpType() ==
                ContainerReplicaOp.PendingOpType.ADD)
            .map(ContainerReplicaOp::getTarget)
            .collect(Collectors.toList());
    excludeList.addAll(pendingReplication);

    /*
    Ensure that target datanodes have enough space to hold a complete
    container.
    */
    final long dataSizeRequired =
        Math.max(replicaCount.getContainer().getUsedBytes(),
            currentContainerSize);
    return placementPolicy.chooseDatanodes(excludeList, null,
        replicaCount.additionalReplicaNeeded(), 0, dataSizeRequired);
}

We ask the placement policy for the required number of nodes. If it cannot provide the required number (eg we want 2, but it can only find 1 spare node), it will throw a SCMException. In that case, we should try again requesting one less node to see if we can partially recover. If that fails, try again with another less node until we reach zero.

If we successfully send a command for some but not all new copies, then we should throw an exception at the end so the container is re-queued on the under-rep queue to be tried again in a short while.

Note the class `MisReplicationHandler.getTargetDatanodes()` currently has logic like this, but it does not throw at the end to trigger a re-queue - HDDS-8337

Issue Links

• links to: GitHub Pull Request #4561