[YARN-10888] [Umbrella] New capacity modes for CS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Szilard Nemeth
• Created: 18/Aug/21 09:17
• Updated: 11/Feb/24 10:59
• Resolved: 25/Jan/24 15:58

Description

Investigate how resource allocation configuration could be more consistent in CapacityScheduler

It would be nice if everywhere where a capacity can be defined could be defined the same way:

• With fixed amounts (e.g. 1 GB memory, 8 vcores, 3 GPU)
• With percentages
  • Percentage of all resources (eg 10% of all memory, vcore, GPU)
  • Percentage per resource type (eg 10% memory, 25% vcore, 50% GPU)
• Allow mixing different modes under one hierarchy but not under the same parent queues.

We need to determine all configuration options where capacities can be defined, and see if it is possible to extend the configuration, or if it makes sense in that case.

Issue Links

• duplicates: YARN-9904 Investigate how resource allocation configuration could be more consistent in CapacityScheduler
• is related to: YARN-11300 [Umbrella] Extract and refactor queue resource calculation + Add a generic way to explain calculations in the logs
• is related to: YARN-11652 [Umbrella] Follow-up after YARN-10888/YARN-10889

Sub-Tasks

• Extend the maximum-capacity property to support Fair Scheduler migration: Resolved
• Support QueueCapacities to use vector based multi resource types, and update absolute related to use first.: Resolved
• Support vector of capacity percentages in Capacity Scheduler configuration: Resolved
• Mixed mode: Allow relative and absolute mode in the same queue hierarchy: Resolved
• Introduce universal configured capacity vector: Resolved
• Centralize queue resource calculation based on CapacityVectors: Resolved
• Replace queue resource calculation logic in updateClusterResource: Resolved
• Extend SchedulerResponse with capacityVector: Resolved
• Support capacity vector for AQCv2 dynamic templates: Resolved
• Create a test set that runs with both Legacy/Uniform queue calculation: Resolved
• Update documentation for the New capacity modes in CS: Resolved
• CapacityScheduler CapacityConfigType changed in legacy queue allocation mode: Resolved
• Fix checkstyle issues after YARN-11520: Resolved