Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496

[Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • capacity scheduler
    • None

    Description

      CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 

      • Only leaf queues can be auto-created
      • A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
      • It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.

       

      • In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
        • This implies that there is no possibility to have both dynamically created and static queues at the same time under root
      • A new queue needs to be created under an existing parent, while the parent already has static queues
      • Nested queue mapping policy, like in the following example: 
      <rule name="nestedUserQueue" create=”true”>
              <rule name="primaryGroup" create="true" />
      </rule>
      • Here two levels of queues may need to be created 

      If an application belongs to user alice (who has the primary_group of engineering), the scheduler checks whether root.engineering exists, if it doesn’t,  it’ll be created. Then scheduler checks whether root.engineering.alice exists, and creates it if it doesn't.

       

      When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.

      Attachments

        Issue Links

        There are no Sub-Tasks for this issue.

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            wangda Wangda Tan
            Votes:
            1 Vote for this issue
            Watchers:
            16 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0h
              0h
              Logged:
              Time Spent - 4h 20m
              4h 20m

              Slack

                Issue deployment