[YARN-10496] [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: capacity scheduler
• Target Version/s: 3.4.0
• Assignee: Unassigned Assign to me
• Reporter: Wangda Tan
• Created: 19/Nov/20 18:41
• Updated: 08/Sep/21 13:55
• Resolved: 08/Sep/21 13:55

Description

CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 

• Only leaf queues can be auto-created
• A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:

• It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.

 

• In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
  • This implies that there is no possibility to have both dynamically created and static queues at the same time under root
• A new queue needs to be created under an existing parent, while the parent already has static queues
• Nested queue mapping policy, like in the following example: 

<rule name="nestedUserQueue" create=”true”>         <rule name="primaryGroup" create="true" /> </rule>

• Here two levels of queues may need to be created 

If an application belongs to user alice (who has the primary_group of engineering), the scheduler checks whether root.engineering exists, if it doesn’t,  it’ll be created. Then scheduler checks whether root.engineering.alice exists, and creates it if it doesn't.

 

When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.

Issue Links

• blocks: YARN-10169 Mixed absolute resource value and percentage-based resource value in CapacityScheduler should fail
• links to: Design document