[HDDS-5655] SCM terminates when allocatecontainer happens on closed pipeline

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SCM HA
• Labels: pull-request-available
• Assignee: Bharat Viswanadham
• Reporter: Bharat Viswanadham
• Created: 23/Aug/21 10:31
• Updated: 25/Aug/21 15:40
• Resolved: 25/Aug/21 15:40

Description

Scenario like this:
1. AllocateContainer selects open pipeline (This point it is open)
2. PipelineActionHandler comes and closes pipeline by calling pipelineManager#closePipeline which calls stateManager#updatePipelineState(pipelineID.getProtobuf(),
HddsProtos.PipelineState.PIPELINE_CLOSED);
3. Now when containerStateManager#addContainer(containerInfo) is called, which calls pipelineManager.addContainerToPipeline(pipelineID, containerID); this will fail, as now pipeline becomes closed due to below piece of code. As containerStateManager#addContainer is replicateCall happens in StateMachine if any exception we terminate SCM.

  void addContainerToPipeline(PipelineID pipelineID, ContainerID containerID)
      throws IOException {
    Preconditions.checkNotNull(pipelineID,
        "Pipeline Id cannot be null");
    Preconditions.checkNotNull(containerID,
        "Container Id cannot be null");

    Pipeline pipeline = getPipeline(pipelineID);
    if (pipeline.isClosed()) {
      throw new IOException(String
          .format("Cannot add container to pipeline=%s in closed state",
              pipelineID));
    }
    pipeline2container.get(pipelineID).add(containerID);
  }

Proposed Solution:
Acquire pipelineManager during allocateContainer to avoid any updates to pipelineState.

Issue Links

• links to: GitHub Pull Request #2569