[FLINK-21075] FLIP-160: Adaptive scheduler

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Assignee: Robert Metzger
• Reporter: Till Rohrmann
• Created: 21/Jan/21 19:58
• Updated: 13/Apr/21 12:12
• Resolved: 07/Apr/21 21:52

Description

The adaptive scheduler is a new scheduler which first declares the required resources and waits for them before deciding on the parallelism with which to execute a job. This behaviour has the benefit that Flink will be able to adjust the job to the available resources and to also make use of resources which arrive later. The details can be found in the FLIP-160.

This ticket serves as the umbrella for all the MVP targeted for Flink 1.13.0 (as described in FLIP-160). Improvements beyond the MVP are captured in there own tickets.

Issue Links

• blocks: FLINK-10407 FLIP-159: Reactive mode
• is depended upon by: FLINK-21510 ExecutionGraph metrics collide on restart
• is depended upon by: FLINK-21450 Add local recovery support to adaptive scheduler
• is depended upon by: FLINK-21439 Adaptive Scheduler: Add support for exception history
• relates to: FLINK-21846 Rethink whether failure of ExecutionGraph creation in Adaptive Scheduler should directly fail the job
• relates to: FLINK-21954 Test failures occur due to the test not waiting for the ExecutionGraph to be created in Adaptive Scheduler
• relates to: FLINK-21964 HA e2e test failed due to not enough resources being available with Adaptive Scheduler
• relates to: FLINK-22255 AdaptiveScheduler improvements/bugs
• relates to: FLINK-21604 Remove adaptive scheduler nightly test profile, replace with more specific testing approach
• links to: FLIP-160
• mentioned in: Page Loading...
• mentioned in: Page Loading...