Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10429

Redesign Flink Scheduling, introducing dedicated Scheduler component

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.0
    • Fix Version/s: None
    • Component/s: Runtime / Coordination
    • Labels:
      None

      Description

      This epic tracks the redesign of scheduling in Flink. Scheduling is currently a concern that is scattered across different components, mainly the ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on the granularity of individual tasks, which make holistic scheduling strategies hard to implement. In this epic we aim to introduce a dedicated Scheduler component that can support use-case like auto-scaling, local-recovery, and resource optimized batch.

      The design for this feature is developed here: https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing

        Attachments

          Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. FLINK-10240 Pluggable scheduling strategy for batch jobs

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-11298 Scheduling job in the unit of concurrent groups

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue FLINK-12369 Implement a region failover strategy based on new FailoverStrategy interfaces

          • Major - Major loss of function.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. FLINK-12002 Adaptive Parallelism of Job Vertex Execution

          • Major - Major loss of function.
          • Open
          supercedes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-10713 RestartIndividualStrategy does not restore state

          • Critical - Crashes, loss of data, severe memory leak.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. FLINK-2250 Backtracking of intermediate results

          • Major - Major loss of function.
          • Closed

            Activity

              People

              • Assignee:
                gjy Gary Yao
                Reporter:
                srichter Stefan Richter
              • Votes:
                3 Vote for this issue
                Watchers:
                45 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 22.5h
                  22.5h