Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496

[Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • capacity scheduler
    • None

    Description

      CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 

      • Only leaf queues can be auto-created
      • A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
      • It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.

       

      • In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
        • This implies that there is no possibility to have both dynamically created and static queues at the same time under root
      • A new queue needs to be created under an existing parent, while the parent already has static queues
      • Nested queue mapping policy, like in the following example: 
      <rule name="nestedUserQueue" create=”true”>
              <rule name="primaryGroup" create="true" />
      </rule>
      • Here two levels of queues may need to be created 

      If an application belongs to user alice (who has the primary_group of engineering), the scheduler checks whether root.engineering exists, if it doesn’t,  it’ll be created. Then scheduler checks whether root.engineering.alice exists, and creates it if it doesn't.

       

      When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              wangda Wangda Tan
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 20m
                  4h 20m