XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Ozone Datanode, SCM
    • Labels:
      None
    • Target Version/s:

      Description

      Apache Ratis supports multi-raft by allowing the same node to be a part of multiple raft groups. The proposal is to allow datanodes to be a part of multiple raft groups. The attached design doc explains the reasons for doing this as well a few initial design decisions.

      Some of the work in this feature also related to HDDS-700 which implements rack-aware container placement for closed containers.

        Attachments

        1. Ozone Multi-Raft Support.pdf
          358 kB
          Siddharth Wagle

          Issue Links

          1.
          Create an interface for pipeline placement policy to support network topologies Sub-task Resolved Li Cheng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          2.
          Add default pipeline placement policy implementation Sub-task Resolved Li Cheng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 5h 20m
          3.
          Add CLI createPipeline Sub-task Resolved Li Cheng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          4.
          Add ability to SCM for creating multiple pipelines with same datanode Sub-task Resolved Li Cheng

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 8.5h
          5.
          Async RATIS pipeline creation and destroy through heartbeat commands Sub-task Resolved Sammi Chen

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 13h
          6.
          Support join multiple pipelines on datanode Sub-task Open Sammi Chen  
          7.
          Support configure more than one raft log storage to host multiple pipelines Sub-task Open Sammi Chen

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 10m
          8.
          Refactor heartbeat reports to report all the pipelines that are open Sub-task Open Li Cheng  
          9.
          Implement a Pipeline scrubber to maintain healthy number of pipelines in a cluster Sub-task Open Unassigned  
          10.
          Add scrubber metrics and pipeline metrics Sub-task Open Unassigned  
          11.
          Fix tests using MiniOzoneCluster for its memory related exceptions Sub-task Resolved Li Cheng  
          12.
          Ensure same datanodes are not a part of multiple pipelines Sub-task Open Li Cheng  
          13.
          Add per pipeline metrics Sub-task Open Unassigned  
          14.
          Add support for storage policies to Pipelines Sub-task Open Unassigned  
          15.
          Improve CLI listPipeline Sub-task Open Unassigned  
          16.
          Multi-raft support on single datanode integration test Sub-task Open Unassigned  
          17.
          Refactor to create pipeline via DN heartbeat response Sub-task Open Unassigned  
          18.
          Add smoke/acceptance test for Pipeline related CLI Sub-task Open Li Cheng  
          19.
          Create SCMPipelineAllocationManager as background thread for pipeline creation Sub-task Open Unassigned  
          20.
          Add new pipeline state “CLOSING” and new CLOSE_PIPELINE_STATUS command Sub-task Open Sammi Chen  
          21.
          Add a srubber thread to detect creation failure pipelines in ALLOCATED state Sub-task Open Unassigned  

            Activity

              People

              • Assignee:
                timmylicheng Li Cheng
                Reporter:
                swagle Siddharth Wagle
              • Votes:
                1 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 29h 20m
                  29h 20m