[FLINK-10404] FLIP-138 - Declarative resource management

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.13.0
• Component/s: Runtime / Coordination
• Assignee: Chesnay Schepler
• Reporter: Till Rohrmann
• Created: 24/Sep/18 09:18
• Updated: 21/Dec/20 14:45
• Resolved: 17/Dec/20 22:00

Description

This is the umbrella issue to track the progress on implementing the declarative resource management described in FLIP-138.

Declarative resource management is a change to Flink's current slot allocation protocol. Instead of letting the JobMaster ask for each slot individually, it will tell the ResourceManager its current need (min, target, max) of slots. Based on that, the ResourceManager will assign available slots or start new TaskExecutors to fulfill the request.

Issue Links

• causes: FLINK-20679 JobMasterTest.testSlotRequestTimeoutWhenNoSlotOffering test failed
• relates to: FLINK-10407 FLIP-159: Reactive mode
• supercedes: FLINK-9956 Push available slots to JobMaster
• links to: Design document
• mentioned in: Page Loading...

Sub-Tasks

• Add feature toggle for declarative resource management: Closed
• Setup Azure build for running core and flink-tests: Closed
• Implement DeclarativeSlotManager: Closed
• Add ResourceRequirement(s): Closed
• Implement declarative SlotPool: Closed
• Remove AllocationIdsExposingResourceManagerGateway: Closed
• Optimize re-assignment of excess resources: Closed
• Support grace period for not enough resources being available at cluster startup: Closed
• DeclarativeSlotManager#suspend can fail with NPE: Closed
• Adjust JobMasterTest to be compatible: Closed