[FLINK-31316] FLIP-291: Externalized Declarative Resource Management

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: 2.0.0
• Component/s: Runtime / Coordination, Runtime / REST
• Assignee: David Morávek
• Reporter: David Morávek
• Created: 03/Mar/23 16:10
• Updated: 10/Jul/24 07:03

Description

This is an umbrella ticket for FLIP-291.

Issue Links

• fixes: FLINK-30773 Add API for rescaling of jobs based on per-vertex parallelism overrides
• relates to: FLINK-31935 The new resource requirements REST API is only available for session clusters
• relates to: FLINK-31996 Chaining operators with different max parallelism prevents rescaling
• relates to: FLINK-31744 Extend Adaptive Scheduler sparse EG to contain maxParallelism
• mentioned in: Page Loading...

Sub-Tasks

• Introduce data structures for managing resource requirements of a job: Resolved
• Add support for documenting additionProperties of the REST API payloads.: Resolved
• Introduce extended Assertj Matchers for completable futures: Resolved
• AdaptiveScheduler is able to handle changes in job resource requirements: Closed
• Allow setting JobResourceRequirements through REST API: Resolved
• Allow setting JobResourceRequirements through DispatcherGateway: Closed
• Introduce integration tests for Externalized Declarative Resource Management: Resolved
• Allow changing per-vertex desired parallelism through WEB UI: Resolved
• AdaptiveScheduler should take lower bound parallelism settings into account: Closed
• Allow setting JobResourceRequirements through JobMasterGateway: Closed
• Extend JobGraphWriter to persist requirements: Closed
• Re-evaluate 'min-parallelism-increase' option: Open
• Requirement validation does not take user-specified or scheduler-generated maxParallelism into account: Closed
• Hide the rescaling buttons for batch jobs in the WEB UI: Open
• Document Externalized Declarative Resource Management: Resolved
• Document Externalized Declarative Resource Management With Chinese: Open