[HDDS-4689] Autoclose pipeline after 'n' containers to distribute pipeline and containers over multiple datanodes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SCM
• Assignee: Nandakumar
• Reporter: Mukul Kumar Singh
• Created: 13/Jan/21 15:32
• Updated: 23/Feb/24 00:05

Description

Currently, Ozone pipelines are long-lived i.e. a pipeline is normally on 3 DN's and it stays there till the pipeline fails or the DN restarts.

This Jira proposes to close the pipeline after let's say 20 containers have been created and then closed on a pipeline. Once the pipeline is closed, the DN's are released and a new pipeline can be created using the set of available nodes during that time.

This patch will hopefully help in increasing the skew of the containers on the DN's and will help in error recovery and data loss cases when 3 DN's are lost.

Issue Links

• relates to: HDDS-6225 Pipelines distribution across datanodes is uneven
• relates to: HDDS-5441 Disallow same set of DNs to be part of multiple pipelines