[FLINK-10407] FLIP-159: Reactive mode

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Assignee: Robert Metzger
• Reporter: Till Rohrmann
• Created: 24/Sep/18 09:21
• Updated: 07/May/21 06:03
• Resolved: 07/Apr/21 18:53

Description

The reactive mode is a new operation mode where a Flink cluster will react to newly available resources (e.g. started by an external service) and make use of them by rescaling the existing job. This ticket acts as an umbrella for the required sub-tasks.

Issue Links

• is blocked by: FLINK-21075 FLIP-160: Adaptive scheduler
• is related to: FLINK-10404 FLIP-138 - Declarative resource management
• relates to: FLINK-21606 TaskManager connected to invalid JobManager leading to TaskSubmissionException
• relates to: FLINK-21883 Introduce cooldown period into adaptive scheduler
• relates to: FLINK-21884 Reduce TaskManager failure detection time
• links to: FLIP-159
• links to: Outdated design document
• mentioned in: Page Loading...
• mentioned in: Page Loading...

Sub-Tasks

• Reactive mode: Introduce execution mode configuration key and check for supported ClusterEntrypoint type: Closed
• Reactive Mode: Change Adaptive Scheduler to set infinite parallelism in JobGraph: Closed
• Reactive Mode: Adjust timeout behavior in adaptive scheduler: Closed
• Document how to use the reactive mode: Closed
• Skip requirement checks if slot report does not indicate any change: Closed
• Do not auto-configure maxParallelism when setting "scheduler-mode: reactive": Closed