Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496

[Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: capacity scheduler
    • Labels:
      None
    • Target Version/s:

      Description

      CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 

      • Only leaf queues can be auto-created
      • A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
      • It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.

       

      • In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
        • This implies that there is no possibility to have both dynamically created and static queues at the same time under root
      • A new queue needs to be created under an existing parent, while the parent already has static queues
      • Nested queue mapping policy, like in the following example: 
      <rule name="nestedUserQueue" create=”true”>
              <rule name="primaryGroup" create="true" />
      </rule>
      • Here two levels of queues may need to be created 

      If an application belongs to user alice (who has the primary_group of engineering), the scheduler checks whether root.engineering exists, if it doesn’t,  it’ll be created. Then scheduler checks whether root.engineering.alice exists, and creates it if it doesn't.

       

      When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.

        Attachments

          Issue Links

          1.
          Implement weight mode in Capacity Scheduler Sub-task Resolved Benjamin Teke  
          2.
          Extend the maximum-capacity property to support Fair Scheduler migration Sub-task Reopened Benjamin Teke  
          3.
          Update queue creation logic to use weight mode and allow the flexible static/dynamic creation Sub-task Resolved Andras Gyori  
          4.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS Sub-task Resolved Szilard Nemeth  
          5.
          CS Flexible Auto Queue Creation RM UIv2 modifications Sub-task Resolved Andras Gyori  
          6.
          To support the mixed mode on different levels(optional) or disabled all. Sub-task Resolved Qi Zhu  
          7.
          Document for Flexible Auto Queue Creation in Capacity Scheduler. Sub-task Open Benjamin Teke  
          8.
          Support multi resource type based weight mode in CS. Sub-task Resolved Qi Zhu  
          9.
          Add weight mode conversion to fs2cs Sub-task Resolved Peter Bacsko  
          10.
          Be able to disable user limit factor for CapacityScheduler Leaf Queue Sub-task Resolved Qi Zhu  
          11.
          Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used Sub-task Resolved Qi Zhu  
          12.
          Make queue placement in CapacityScheduler compliant with auto-queue-placement Sub-task Resolved Gergely Pollák  
          13.
          Support Auto Queue Creation template configurations Sub-task Resolved Andras Gyori  
          14.
          Refactor CS queue initialization to simplify weight mode calculation Sub-task Open Benjamin Teke  
          15.
          Refactor dynamic queue handling logic Sub-task Resolved Andras Gyori  
          16.
          Enhance placement rule conversion in fs2cs in weight mode and enable it by default Sub-task Resolved Peter Bacsko  
          17.
          Fix the FindBugs warning introduced in YARN-10506 Sub-task Resolved Gergely Pollák  
          18.
          Update Capacity Scheduler documentation about JSON-based placement mapping Sub-task Patch Available Peter Bacsko  
          19.
          Automatically convert placement rules in fs2cs Sub-task Resolved Peter Bacsko  
          20.
          Fix Auto Queue Creation parent handling Sub-task Resolved Andras Gyori  
          21.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues Sub-task Resolved Szilard Nemeth  
          22.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues Sub-task Resolved Szilard Nemeth  
          23.
          Increase the test coverage for CSMappingPlacementRule Sub-task Open Gergely Pollák  
          24.
          Increase the test coverage for validation in MappingRuleValidationHelper Sub-task Open Gergely Pollák  
          25.
          Move the functionality of MappingRuleValidationHelper to CSQueueManager Sub-task Open Gergely Pollák  
          26.
          Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. Sub-task Resolved Qi Zhu  
          27.
          Fix TestCapacitySchedulerAutoCreatedQueueBase witch related absolute caculation loss. Sub-task Patch Available Qi Zhu  
          28.
          Support QueueCapacities to use vector based multi resource types, and update absolute related to use first. Sub-task Patch Available Qi Zhu  
          29.
          Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled Sub-task Resolved Andras Gyori  
          30.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information Sub-task Resolved Benjamin Teke  
          31.
          fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents Sub-task Resolved Peter Bacsko  
          32.
          Convert root queue in fs2cs weight mode conversion Sub-task Resolved Benjamin Teke  
          33.
          Support auto queue creation without mapping rules Sub-task Resolved Andras Gyori  
          34.
          Add queue-mappings-override.enable property in FS2CS conversions Sub-task Resolved Andras Gyori  
          35.
          Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue) Sub-task Resolved Qi Zhu  
          36.
          Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion Sub-task Resolved Benjamin Teke  
          37.
          Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name Sub-task Resolved Andras Gyori  
          38.
          fs2cs: parentQueue for certain placement rules are not set during conversion Sub-task Resolved Peter Bacsko  
          39.
          Fix preemption policy to exclude childless ParentQueues Sub-task Resolved Andras Gyori  
          40.
          Support max queues limit configuration in new auto created queue, consistent with old auto created. Sub-task Resolved Qi Zhu  
          41.
          Extend logging to give more information about weight mode Sub-task Resolved Benjamin Teke  
          42.
          Make maximum depth allowed configurable. Sub-task Patch Available Qi Zhu  
          43.
          CS Auto Queue creation should reject submissions with empty path parts Sub-task Resolved Gergely Pollák  
          44.
          Queueinfo related capacity, should adjusted to weight mode. Sub-task Resolved Qi Zhu  
          45.
          Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Sub-task Resolved Qi Zhu  
          46.
          Refactor the max app related update, and fix maxApplications update error when add new queues. Sub-task Resolved Qi Zhu  
          47.
          Fix queue state related update for auto created queue. Sub-task Resolved Qi Zhu  
          48.
          TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario Sub-task Open Benjamin Teke  
          49.
          Limit queue creation depth relative to its first static parent Sub-task Resolved Andras Gyori  
          50.
          We should make max application per queue to support node label. Sub-task Resolved Andras Gyori  
          51.
          Fix the spelling errors in TestCapacitySchedulerWeightMode about allocation. Sub-task Resolved Qi Zhu  
          52.
          fs2cs should generate auto-created queue deletion properties Sub-task Resolved Qi Zhu  
          53.
          Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode Sub-task Resolved Qi Zhu  
          54.
          Fix the findbugs issues in extractFloatValueFromWeightConfig. Sub-task Resolved Qi Zhu

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          55.
          Remove dangling dynamic queues on reinitialization Sub-task Resolved Andras Gyori  
          56.
          ParentQueue does not validate the queue on removal Sub-task Resolved Andras Gyori  
          57.
          Extend /scheduler endpoint with template properties Sub-task Resolved Andras Gyori  
          58.
          Allow definition of auto queue template properties in root Sub-task Resolved Andras Gyori  
          59.
          CS Flexible AQC: Add separate parent and leaf template property. Sub-task Resolved Andras Gyori  
          60.
          Fix Auto Queue template to properly set all configuration properties Sub-task Resolved Andras Gyori  
          61.
          Parents node labels are incorrectly added to child queues in weight mode Sub-task Resolved Benjamin Teke  

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                wangda Wangda Tan
              • Votes:
                1 Vote for this issue
                Watchers:
                14 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m