Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-10429

Redesign Flink Scheduling, introducing dedicated Scheduler component

          Details

          • Type: New Feature
          • Status: Open
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: 1.7.0
          • Fix Version/s: None
          • Component/s: Runtime / Coordination
          • Labels:
            None

            Description

            This epic tracks the redesign of scheduling in Flink. Scheduling is currently a concern that is scattered across different components, mainly the ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on the granularity of individual tasks, which make holistic scheduling strategies hard to implement. In this epic we aim to introduce a dedicated Scheduler component that can support use-case like auto-scaling, local-recovery, and resource optimized batch.

            The design for this feature is developed here: https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing

              Attachments

                Issue Links

                blocks

                New Feature - A new feature of the product, which has yet to be developed. FLINK-10240 Pluggable scheduling strategy for batch jobs

                • Major - Major loss of function.
                • Open

                Improvement - An improvement or enhancement to an existing feature or task. FLINK-11298 Scheduling job in the unit of concurrent groups

                • Major - Major loss of function.
                • Open
                relates to

                Sub-task - The sub-task of the issue FLINK-12369 Implement a region failover strategy based on new FailoverStrategy interfaces

                • Major - Major loss of function.
                • Closed

                Improvement - An improvement or enhancement to an existing feature or task. FLINK-12002 Adaptive Parallelism of Job Vertex Execution

                • Major - Major loss of function.
                • Open
                supercedes

                Bug - A problem which impairs or prevents the functions of the product. FLINK-10713 RestartIndividualStrategy does not restore state

                • Critical - Crashes, loss of data, severe memory leak.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. FLINK-2250 Backtracking of intermediate results

                • Major - Major loss of function.
                • Closed
                1.
                		 Make access to ExecutionGraph single threaded from JobMaster main thread Sub-task Closed Stefan Richter
                2.
                		 Extract scheduling-related code from Executions Sub-task Open Stefan Richter  
                3.
                		 Extract scheduling-related code from SlotPool Sub-task Closed Stefan Richter
                4.
                		 Introduce bulk/group-aware scheduling Sub-task Open Stefan Richter  
                5.
                		 Stepwise creation of the ExecutionGraph sub-structures Sub-task Open Stefan Richter  
                6.
                		 Introduce SchedulingStrategy interface Sub-task Closed shuai.xu
                7.
                		 Implement Eager Scheduling Strategy Sub-task Closed shuai.xu
                8.
                		 Implement Lazy Scheduling Strategy Sub-task Closed BoWang
                9.
                		 Remove getExecutionGraph() from JobMaster Sub-task Closed Gary Yao
                10.
                		 Introduce Scheduler interface and adapt ExecutionGraph to it Sub-task Closed Gary Yao
                11.
                		 Temporarily disable CLI command for rescaling Sub-task Closed Gary Yao
                12.
                		 Add CLI command for rescaling Sub-task Open Unassigned  
                13.
                		 Implement ExecutionSlotAllocator Sub-task Closed shuai.xu
                14.
                		 Implement ExecutionFailureHandler Sub-task Closed Zhu Zhu
                15.
                		 Implement ExecutionGraph to SchedulingTopology Adapter Sub-task Closed BoWang
                16.
                		 Add SchedulerNG Stub Implementation Sub-task Closed Gary Yao
                17.
                		 Finalize SchedulerNG Stub Implementation Sub-task In Progress Gary Yao  
                18.
                		 Add getVertex/ResultPartitionOrThrow(ExecutionVertexID/IntermediateResultPartitionID) to SchedulingTopology Sub-task Closed BoWang
                19.
                		 Implement FixedDelayRestartBackoffTimeStrategy Sub-task Closed BoWang
                20.
                		 Implement FailureRateRestartBackoffTimeStrategy Sub-task Closed BoWang
                21.
                		 Implement RestartBackoffTimeStrategyFactoryLoader Sub-task In Progress Zhu Zhu
                22.
                		 Implement ExecutionGraph to InputsLocationsRetriever Adapter Sub-task Closed shuai.xu

                  Activity

                    People

                    • Assignee:
                      gjy Gary Yao
                      Reporter:
                      srichter Stefan Richter
                    • Votes:
                      2 Vote for this issue
                      Watchers:
                      41 Start watching this issue

                      Dates

                      • Created:
                        Updated:

                        Time Tracking

                        Estimated:
                        Original Estimate - Not Specified
                        Not Specified
                        Remaining:
                        Remaining Estimate - 0h
                        0h
                        Logged:
                        Time Spent - 5h 40m
                        5h 40m