[HDDS-1564] Ozone multi-raft support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Ozone Datanode, SCM
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Assignee: Li Cheng
• Reporter: Siddharth Wagle
• Created: 21/May/19 04:09
• Updated: 25/Feb/20 18:06
• Resolved: 19/Feb/20 06:39

Description

Apache Ratis supports multi-raft by allowing the same node to be a part of multiple raft groups. The proposal is to allow datanodes to be a part of multiple raft groups. The attached design doc explains the reasons for doing this as well a few initial design decisions.

Some of the work in this feature also related to HDDS-700 which implements rack-aware container placement for closed containers.

Issue Links

• Blocked: HDDS-2010 PipelineID management for multi-raft, in SCM or in datanode?
• breaks: HDDS-3038 TestRatisPipelineLeader fails since we no longer wait for leader in the HealthyPipelineSafeModeExitRule
• is related to: HDDS-698 Support Topology Awareness for Ozone
• relates to: HDDS-2034 Async RATIS pipeline creation and destroy through heartbeat commands
• links to: GitHub Pull Request #538

Sub-Tasks

• Create an interface for pipeline placement policy to support network topologies: Resolved
• Add default pipeline placement policy implementation: Resolved
• Add CLI createPipeline: Resolved
• Add ability to SCM for creating multiple pipelines with same datanode: Resolved
• Fix tests using MiniOzoneCluster for its memory related exceptions: Resolved
• Implement a Pipeline scrubber to clean up non-OPEN pipeline: Resolved
• Refactor heartbeat reports to report all the pipelines that are open: Resolved
• Add scrubber metrics and pipeline metrics: Resolved
• Fix createPipeline and make createPipeline CLI message based.: Resolved
• Multi-raft support on single datanode integration test: Resolved
• Refactor to create pipeline via DN heartbeat response: Resolved
• Create SCMPipelineAllocationManager as background thread for pipeline creation: Resolved
• Support join multiple pipelines on datanode: Resolved
• Add a srubber thread to detect creation failure pipelines in ALLOCATED state: Resolved
• Average out pipeline allocation on datanodes and add metrics/test: Resolved
• Implement datanode level CLI to reveal pipeline info: Resolved
• Support configure more than one raft log storage to host multiple pipelines: Resolved
• Handle pipeline creation failure in different way when it exceeds pipeline limit: Resolved
• Add smoke/acceptance test for createPipeline CLI and datanode list CLI: Resolved
• Better management for pipeline creation limitation: Resolved
• Rename multi-raft ozone default configs for better understanding.: Resolved
• Fix Pipeline#nodeIdsHash collision issue: Resolved
• Add fall-back protection for rack awareness in pipeline creation: Resolved
• Fix CI test failure for TestSCMNodeManager: Resolved