[FLINK-35549] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.20.0
• Fix Version/s: 2.0-preview
• Component/s: Runtime / Checkpointing, Runtime / Coordination

Release Note

FLIP-461 enables the user to synchronize checkpointing and rescaling in the AdaptiveScheduler. jobmanager.adaptive-scheduler.max-delay-for-scale-trigger and jobmanager.adaptive-scheduler.scale-on-failed-checkpoints-count configuration parameters were introduced to make this behavior configurable. See Flink's configuration documentation for further details.

• Assignee: Matthias Pohl
• Reporter: Matthias Pohl
• Created: 07/Jun/24 09:14
• Updated: 02/Sep/24 09:35
• Resolved: 05/Jul/24 15:45

Description

This is the umbrella issue for implementing FLIP-461

Sub-Tasks

• Introduce new component RescaleManager: Resolved
• Introduces RescaleManager#onTrigger endpoint: Resolved
• Move CheckpointStatsTracker out of ExecutionGraph into Scheduler: Resolved
• Integrate newly added trigger interface with checkpointing: Resolved