[FLINK-15626] Remove legacy scheduler

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.11.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Labels: Umbrella
• Assignee: Zhu Zhu
• Reporter: Zhu Zhu
• Created: 17/Jan/20 03:24
• Updated: 27/Apr/21 18:07
• Resolved: 25/Jan/21 05:41

Description

This umbrella ticket is to track the tickets to remove the legacy scheduler and related components.
So that we can have a much cleaner scheduler framework which significantly simplifies our next development work on job scheduling.

Here's a rough plan:
1. remove LegacyScheduler class

• related tests should also be removed, including AlsoRunWithLegacyScheduler which is used to enable integration tests for legacy scheduler

2. remove legacy failover strategy AdaptedRestartPipelinedRegionStrategyNG and RestartIndividualStrategy. These implementations and their tests heavily relies on the legacy scheduling logics in ExecutionGraph so removing them can make #3 much easier.

3. remove legacy scheduling/failover logics in ExecutionGraph

• note that many tests still relies on these logics(e.g. ExecutionGraph#scheduleForExecution()), we may need to retain and rewrite tests which are still needed. So before removing the legacy logics, it's better to cleanup all the tests relying on the legacy scheduling first, i.e. remove tests that are no needed and rewrite the tests which are still needed.

4. remove legacy FailoverStrategy and RestartStrategy

5. Factoring out the scheduling logics in ExecutionGraph to make it a plain data structure. This can be a long term goal though. There can be several pieces and we may need to examine them one by one.

Issue Links

• blocks: FLINK-10429 Redesign Flink Scheduling, introducing dedicated Scheduler component
• relates to: FLINK-20589 Remove old scheduling strategies
• mentioned in: Page Loading...