[HDDS-8919] Allow EC pipelines to be created and then added to pipeline manager in two steps

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 23/Jun/23 16:09
• Updated: 26/Jun/23 21:04
• Resolved: 26/Jun/23 21:04

Description

As it stands, the method:

 PipelineManager.createPipeline(ReplicationConfig replicationConfig,
      List<DatanodeDetails> excludedNodes, List<DatanodeDetails> favoredNodes)

Is used to create a new pipeline and add it to the Pipeline manager. This creation and adding is all performed under a single lock The somewhat expensive node selection process from the placement policy is performed under the lock.

For Ratis pipelines, this could be important, as pipelines are created and limited based on the number of existing pipelines on the nodes.

EC pipelines do not have such a limitation, and they also need to be created much more frequently due to their short lived nature.

This PR changes the pipeline manager to have a buildPipeline Method, which calls the placement policy and creates the new pipeline object without any locks.

Then there is an addPipeline method to add the created pipeline to the pipelineManager under the lock.

This will allow for the WritableECPipelineProvider to build pipelines without blocking other threads and then add them to the manager under a lock, reducing the amount of work performed under a lock. This change would need to be made in another PR.

Issue Links

• links to: GitHub Pull Request #4968