[FLINK-14106] Make SlotManager pluggable

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Done
• Affects Version/s: 1.9.0
• Fix Version/s: 1.11.0
• Component/s: Runtime / Coordination
• Labels: stale-assigned
• Assignee: Xintong Song
• Reporter: Xintong Song
• Created: 17/Sep/19 14:11
• Updated: 16/Apr/21 11:11
• Resolved: 16/Apr/21 11:11

Description

As we are enabling fine grained resource management in 1.10, we can have various resource scheduling strategies. Such strategies generally should make the following three decisions.

• When to launch new / release existing TMs? (How many TMs)
• What and how many resources should TMs be started with?
• How to allocate between slot requests and TM resources?

We may want to make above decisions differently in different scenarios (active/reactive mode, perjob/session mode, etc.). Therefore, we propose to make the scheduling strategies pluggable.

We propose to make the following changes:

• Make SlotManager an interface, and implements it differently for different strategies strategies.
• Modify ResourceManager-SlotManager interfaces to cover all the three decisions mentioned above in SlotManager. In particular, SlotManager needs to allocate TM resources instead of slot resources from ResourceActions.

Issue Links

• blocks: FLINK-14187 FLIP-56 Dynamic Slot Allocation
• is related to: FLINK-19324 Map requested/allocated containers with priority on YARN
• is required by: FLINK-5791 Support an optimal matching based slot manager for flip6 yarn mode
• relates to: FLINK-17061 Unset process/flink memory size from configuration once dynamic worker resource is activated.
• links to: Proposed Changes