[YARN-9936] Support vector of capacity percentages in Capacity Scheduler configuration

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: capacity scheduler
• Target Version/s: 3.4.0
• Assignee: Andras Gyori
• Reporter: Zoltan Siegl
• Created: 28/Oct/19 09:27
• Updated: 18/Aug/21 09:24

Description

Currently, the Capacity Scheduler queue configuration supports two ways to set queue capacity.

• In percentage of all available resources as a float ( eg. 25.0 ) means 25% of the resources of its parent queue for all resource types equally (eg. 25% of all memory, 25% of all CPU cores, and 25% of all available GPU in the cluster) The percentages of all queues has to add up to 100%.
• In an absolute amount of resources ( e.g. memory=4GB,vcores=20,yarn.io/gpu=4 ). The amount of all resources in the queues has to be less than or equal to all resources in the cluster.Actually, the above is not supported, we only support memory and vcores now in absolute mode, we should extend in  YARN-10503.

Apart from these two already existing ways, there is a demand to add capacity percentage of each available resource type separately. (eg. memory=20%,vcores=40%,yarn.io/gpu=100%).
At the same time, a similar concept should be included with queues maximum-capacity as well.

Issue Links

• Dependent: YARN-10887 Investigation: Decouple capacity and max-capacity modes
• is blocked by: YARN-10759 Encapsulate queue config modes
• is duplicated by: YARN-9891 Capacity scheduler: enhance capacity / maximum-capacity setting
• is duplicated by: YARN-9893 Capacity scheduler: enhance leaf-queue-template capacity / maximum-capacity setting
• is related to: YARN-10505 Extend the maximum-capacity property to support Fair Scheduler migration