[FLINK-16605] Add max limitation to the total number of slots

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available

Release Note

With FLINK-16605, user could define the maximum number of slots through "slotmanager.number-of-slots.max" that the Flink cluster allocates.

This is meant for limiting the resource consumption for batch workloads. It is not recommended to configure this option for streaming workloads, which may fail if there are not enough slots. Note that this configuration option does not take effect for standalone clusters, where how many slots are allocated is not controlled by Flink.

• Assignee: Yangze Guo
• Reporter: Yangze Guo
• Created: 16/Mar/20 02:45
• Updated: 30/Nov/21 20:37
• Resolved: 22/May/20 08:28

Description

As discussed in FLINK-15527 and FLINK-15959, we propose to add the max limit to the total number of slots.

To be specific:

• Introduce "cluster.number-of-slots.max" configuration option with default value MAX_INT
• Make the SlotManager respect the max number of slots, when exceeded, it would not allocate resource anymore.

Issue Links

• is duplicated by: FLINK-15527 can not control the number of container on yarn single job module
• is related to: FLINK-15527 can not control the number of container on yarn single job module
• is related to: FLINK-15959 Add min number of slots configuration to limit total number of slots
• links to: GitHub Pull Request #11615