[HDDS-8698] EC: Avoid unbounded pipeline creation if all existing pipelines don't meet criteria

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Stephen O'Donnell
• Reporter: Stephen O'Donnell
• Created: 25/May/23 13:24
• Updated: 05/Jun/23 19:25
• Resolved: 05/Jun/23 19:25

Description

Currently, if the pipeline limit is not reached, when a new block is requested, a new pipeline is allocated and returned to the client.

If the pipeline limit is reached, we attempt to return a pipeline from the existing set.

If none of those pipelines meet the criteria, then we fall back to creating another pipeline. This could result in unbounded pipeline growth if a client is having problems.

We should therefore limit the pipelines that can be created via this exceptional case.

For example, if we close some pipelines when searching the existing list (due to them being full), we know we have freed up some capacity to create a new pipeline.

Also, or alternative, we could allow the limit to be exceeded by some controlled factor - eg 10% and if the limit is beyond that, return a failure to the client.

Issue Links

• links to: GitHub Pull Request #4829