Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10429

Redesign Flink Scheduling, introducing dedicated Scheduler component

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.0
    • Fix Version/s: None
    • Component/s: Runtime / Coordination
    • Labels:
      None

      Description

      This epic tracks the redesign of scheduling in Flink. Scheduling is currently a concern that is scattered across different components, mainly the ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on the granularity of individual tasks, which make holistic scheduling strategies hard to implement. In this epic we aim to introduce a dedicated Scheduler component that can support use-case like auto-scaling, local-recovery, and resource optimized batch.

      The design for this feature is developed here: https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gjy Gary Yao
                Reporter:
                srichter Stefan Richter
              • Votes:
                3 Vote for this issue
                Watchers:
                45 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 22.5h
                  22.5h