[YARN-5881] [Umbrella] Enable configuration of queue capacity in terms of absolute resources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Assignee: Sunil G
• Reporter: Sean Po
• Created: 15/Nov/16 00:01
• Updated: 28/Nov/20 16:10
• Resolved: 06/Apr/18 18:33

Description

Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion to cluster capacity. In the context of Yarn being used as a public cloud service, it makes more sense if queues can be configured absolutely. This will allow administrators to set usage limits more concretely and simplify customer expectations for cluster allocation.

Issue Links

• causes: YARN-9351 user can't use total resources of one partition even when yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent is set to 100
• incorporates: YARN-10503 Support queue capacity in terms of absolute resources with custom resourceType.
• relates to: YARN-8945 Calculation of maximum applications should respect specified and global maximum applications for absolute resource

Sub-Tasks

• Support to add min/max resource configuration for a queue: Resolved
• UI and metrics changes related to absolute resource configuration: Resolved
• Compute effectiveCapacity per each resource vector: Resolved
• Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee: Resolved
• Max applications calculation per queue has to be retrospected with absolute resource support: Resolved
• CapacityScheduler test cases cleanup post YARN-5881: Resolved
• Merge work for YARN-5881: Resolved
• Documentation for absolute resource support in Capacity Scheduler: Resolved
• Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring: Resolved
• Use queue-path.capacity/maximum-capacity to specify CapacityScheduler absolute min/max resources: Resolved
• Cleanup to fix checkstyle issues of YARN-5881 branch: Resolved
• NPE in scheduler UI when max-capacity is not configured: Resolved