[FLINK-21450] Add local recovery support to adaptive scheduler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.18.0
• Component/s: Runtime / Coordination
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Roman Khachatryan
• Reporter: Robert Metzger
• Created: 23/Feb/21 07:34
• Updated: 25/Mar/24 16:13
• Resolved: 02/Mar/23 21:42

Description

local recovery means that, on a failure, we are able to re-use the state in a taskmanager, instead of loading it again from distributed storage (which means the scheduler needs to know where which state is located, and schedule tasks accordingly).

Adaptive Scheduler is currently not respecting the location of state, so failures require the re-loading of state from the distributed storage.

Adding this feature will allow us to enable the Local recovery and sticky scheduling end-to-end test for adaptive scheduler again.

Issue Links

• causes: FLINK-34409 Increase test coverage for AdaptiveScheduler
• depends upon: FLINK-21075 FLIP-160: Adaptive scheduler
• relates to: FLINK-21737 FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover fails
• relates to: FLINK-32816 Remove the local recovery limitation for adaptive scheduler
• relates to: FLINK-31261 Make AdaptiveScheduler aware of the (local) state size
• links to: GitHub Pull Request #21981