Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7117

[Umbrella] Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacity scheduler
    • Labels:
      None

      Description

      Currently Capacity Scheduler doesn't support auto creation of queues when doing queue mapping. We saw more and more use cases which has complex queue mapping policies configured to handle application to queues mapping.

      The most common use case of CapacityScheduler queue mapping is to create one queue for each user/group. However update capacity-scheduler.xml and RMAdmin:refreshQueues needs to be done when new user/group onboard. One of the option to solve the problem is automatically create queues when new user/group arrives.

        Attachments

        1. YARN-7117_Workflow.pdf
          133 kB
          Suma Shivaprasad
        2. YARN-7117.Capacity.Scheduler.Support.Auto.Creation.Of.Leaf.Queue.pdf
          241 kB
          Wangda Tan
        3. YARN-7117.poc.1.patch
          173 kB
          Suma Shivaprasad
        4. YARN-7117.poc.patch
          169 kB
          Suma Shivaprasad
        1.
        Merge code paths for Reservation/Plan queues and Auto Created queues Sub-task Resolved Suma Shivaprasad Actions
        2.
        CapacityScheduler: Allow auto leaf queue creation after queue mapping Sub-task Resolved Suma Shivaprasad Actions
        3.
        YARN UI changes to depict auto created queues Sub-task Resolved Suma Shivaprasad Actions
        4.
        Implement Framework and policy for capacity management of auto created queues Sub-task Resolved Suma Shivaprasad Actions
        5.
        Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues Sub-task Resolved Suma Shivaprasad Actions
        6.
        Add support for Node Labels on Auto Created Leaf Queue Template Sub-task Resolved Suma Shivaprasad Actions
        7.
        Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management Sub-task Resolved Suma Shivaprasad Actions
        8.
        Documentation for auto queue creation feature and related configurations Sub-task Resolved Suma Shivaprasad Actions
        9.
        Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues Sub-task Resolved Suma Shivaprasad Actions
        10.
        Unit tests related to preemption for auto created leaf queues feature Sub-task Resolved Suma Shivaprasad Actions
        11.
        Queue Management scheduling edit policy class needs to be configured dynamically Sub-task Patch Available Suma Shivaprasad Actions
        12.
        Handle recovery of applications in case of auto-created leaf queue mapping Sub-task Resolved Suma Shivaprasad Actions
        13.
        Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue Sub-task Resolved Suma Shivaprasad Actions
        14.
        Documentation changes for auto creation of Leaf Queues with node label Sub-task Resolved Suma Shivaprasad Actions
        15.
        Yarn UI2 : Changes to depict Auto Created leaf Queues/Managed Queues differently from other queues Sub-task Open Unassigned Actions
        16.
        Logging improvements for better debuggability Sub-task Resolved Suma Shivaprasad Actions

          Activity

            People

            • Assignee:
              suma.shivaprasad Suma Shivaprasad
              Reporter:
              leftnoteasy Wangda Tan

              Dates

              • Created:
                Updated:

                Issue deployment