Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7117

[Umbrella] Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • capacity scheduler
    • None

    Description

      Currently Capacity Scheduler doesn't support auto creation of queues when doing queue mapping. We saw more and more use cases which has complex queue mapping policies configured to handle application to queues mapping.

      The most common use case of CapacityScheduler queue mapping is to create one queue for each user/group. However update capacity-scheduler.xml and RMAdmin:refreshQueues needs to be done when new user/group onboard. One of the option to solve the problem is automatically create queues when new user/group arrives.

      Attachments

        1. YARN-7117_Workflow.pdf
          133 kB
          Suma Shivaprasad
        2. YARN-7117.Capacity.Scheduler.Support.Auto.Creation.Of.Leaf.Queue.pdf
          241 kB
          Wangda Tan
        3. YARN-7117.poc.1.patch
          173 kB
          Suma Shivaprasad
        4. YARN-7117.poc.patch
          169 kB
          Suma Shivaprasad
        1.
        		 Merge code paths for Reservation/Plan queues and Auto Created queues Sub-task Resolved Suma Shivaprasad
        2.
        		 CapacityScheduler: Allow auto leaf queue creation after queue mapping Sub-task Resolved Suma Shivaprasad
        3.
        		 YARN UI changes to depict auto created queues Sub-task Resolved Suma Shivaprasad
        4.
        		 Implement Framework and policy for capacity management of auto created queues Sub-task Resolved Suma Shivaprasad
        5.
        		 Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues Sub-task Resolved Suma Shivaprasad
        6.
        		 Add support for Node Labels on Auto Created Leaf Queue Template Sub-task Resolved Suma Shivaprasad
        7.
        		 Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management Sub-task Resolved Suma Shivaprasad
        8.
        		 Documentation for auto queue creation feature and related configurations Sub-task Resolved Suma Shivaprasad
        9.
        		 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues Sub-task Resolved Suma Shivaprasad
        10.
        		 Unit tests related to preemption for auto created leaf queues feature Sub-task Resolved Suma Shivaprasad
        11.
        		 Queue Management scheduling edit policy class needs to be configured dynamically Sub-task Patch Available Suma Shivaprasad
        12.
        		 Handle recovery of applications in case of auto-created leaf queue mapping Sub-task Resolved Suma Shivaprasad
        13.
        		 Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue Sub-task Resolved Suma Shivaprasad
        14.
        		 Documentation changes for auto creation of Leaf Queues with node label Sub-task Resolved Suma Shivaprasad
        15.
        		 Yarn UI2 : Changes to depict Auto Created leaf Queues/Managed Queues differently from other queues Sub-task Open Unassigned
        16.
        		 Logging improvements for better debuggability Sub-task Resolved Suma Shivaprasad

        Activity

          People

            suma.shivaprasad Suma Shivaprasad
            leftnoteasy Wangda Tan
            Votes:
            0 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

              Created:
              Updated: