[HDDS-11099] Ozone will have only one pipeline for ratis keys

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: SCM
• Assignee: Swaminathan Balachandran
• Reporter: Swaminathan Balachandran
• Created: 04/Jul/24 03:26
• Updated: 04/Jul/24 03:27

Description

If ozone.replication.type is EC, background pipeline creator will not create any pipeline for ratis. When a block allocate request comes to the SCM, the SCM will try to find an existing container to allocate the block. If it is not able to find a suitable container this will lead to creating a ratis container on the fly. While trying to create a container it will look for existing pipelines in the system and if there is no pipeline, a pipeline will be created and if there is only a single pipeline in the system, the same pipeline will be used to create a container over and over again for all blocks. This will happen till the DN doesn't run out of space and another pipeline would be created. But this would lead to writes getting choked to just 3 nodes at a time. 

https://github.com/apache/ozone/blob/c1e9e9180960777ac99e6483515e8b76e585f5d0/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/WritableRatisContainerProvider.java#L163

cc umamaheswararao adoroszlai