Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496

[Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • capacity scheduler
    • None

    Description

      CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 

      • Only leaf queues can be auto-created
      • A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
      • It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.

       

      • In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
        • This implies that there is no possibility to have both dynamically created and static queues at the same time under root
      • A new queue needs to be created under an existing parent, while the parent already has static queues
      • Nested queue mapping policy, like in the following example: 
      <rule name="nestedUserQueue" create=”true”>
              <rule name="primaryGroup" create="true" />
      </rule>
      • Here two levels of queues may need to be created 

      If an application belongs to user alice (who has the primary_group of engineering), the scheduler checks whether root.engineering exists, if it doesn’t,  it’ll be created. Then scheduler checks whether root.engineering.alice exists, and creates it if it doesn't.

       

      When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.

      Attachments

        Issue Links

          1.
          Implement weight mode in Capacity Scheduler Sub-task Resolved Benjamin Teke  
          2.
          Update queue creation logic to use weight mode and allow the flexible static/dynamic creation Sub-task Resolved Andras Gyori  
          3.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS Sub-task Resolved Szilard Nemeth  
          4.
          CS Flexible Auto Queue Creation RM UIv2 modifications Sub-task Resolved Andras Gyori  
          5.
          To support the mixed mode on different levels(optional) or disabled all. Sub-task Resolved Qi Zhu  
          6.
          Document for Flexible Auto Queue Creation in Capacity Scheduler Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          7.
          Support multi resource type based weight mode in CS. Sub-task Resolved Qi Zhu  
          8.
          Add weight mode conversion to fs2cs Sub-task Resolved Peter Bacsko  
          9.
          Be able to disable user limit factor for CapacityScheduler Leaf Queue Sub-task Resolved Qi Zhu  
          10.
          Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used Sub-task Resolved Qi Zhu  
          11.
          Make queue placement in CapacityScheduler compliant with auto-queue-placement Sub-task Resolved Gergely Pollák  
          12.
          Support Auto Queue Creation template configurations Sub-task Resolved Andras Gyori  
          13.
          Refactor dynamic queue handling logic Sub-task Resolved Andras Gyori  
          14.
          Enhance placement rule conversion in fs2cs in weight mode and enable it by default Sub-task Resolved Peter Bacsko  
          15.
          Fix the FindBugs warning introduced in YARN-10506 Sub-task Resolved Gergely Pollák  
          16.
          Update Capacity Scheduler documentation with JSON-based placement mapping Sub-task Resolved Benjamin Teke

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h
          17.
          Automatically convert placement rules in fs2cs Sub-task Resolved Peter Bacsko  
          18.
          Fix Auto Queue Creation parent handling Sub-task Resolved Andras Gyori  
          19.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues Sub-task Resolved Szilard Nemeth  
          20.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues Sub-task Resolved Szilard Nemeth  
          21.
          Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. Sub-task Resolved Qi Zhu  
          22.
          Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled Sub-task Resolved Andras Gyori  
          23.
          CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information Sub-task Resolved Benjamin Teke  
          24.
          fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents Sub-task Resolved Peter Bacsko  
          25.
          Convert root queue in fs2cs weight mode conversion Sub-task Resolved Benjamin Teke  
          26.
          Support auto queue creation without mapping rules Sub-task Resolved Andras Gyori  
          27.
          Add queue-mappings-override.enable property in FS2CS conversions Sub-task Resolved Andras Gyori  
          28.
          Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue) Sub-task Resolved Qi Zhu  
          29.
          Add auto-create-v2-enabled to root queue in fs2cs weight mode conversion Sub-task Resolved Benjamin Teke  
          30.
          Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name Sub-task Resolved Andras Gyori  
          31.
          fs2cs: parentQueue for certain placement rules are not set during conversion Sub-task Resolved Peter Bacsko  
          32.
          Fix preemption policy to exclude childless ParentQueues Sub-task Resolved Andras Gyori  
          33.
          Support max queues limit configuration in new auto created queue, consistent with old auto created. Sub-task Resolved Qi Zhu  
          34.
          Extend logging to give more information about weight mode Sub-task Resolved Benjamin Teke  
          35.
          CS Auto Queue creation should reject submissions with empty path parts Sub-task Resolved Gergely Pollák  
          36.
          Queueinfo related capacity, should adjusted to weight mode. Sub-task Resolved Qi Zhu  
          37.
          Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Sub-task Resolved Qi Zhu  
          38.
          Refactor the max app related update, and fix maxApplications update error when add new queues. Sub-task Resolved Qi Zhu  
          39.
          Fix queue state related update for auto created queue. Sub-task Resolved Qi Zhu  
          40.
          Limit queue creation depth relative to its first static parent Sub-task Resolved Andras Gyori  
          41.
          We should make max application per queue to support node label. Sub-task Resolved Andras Gyori  
          42.
          Fix the spelling errors in TestCapacitySchedulerWeightMode about allocation. Sub-task Resolved Qi Zhu  
          43.
          fs2cs should generate auto-created queue deletion properties Sub-task Resolved Qi Zhu  
          44.
          Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode Sub-task Resolved Qi Zhu  
          45.
          Fix the findbugs issues in extractFloatValueFromWeightConfig. Sub-task Resolved Qi Zhu

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          46.
          Remove dangling dynamic queues on reinitialization Sub-task Resolved Andras Gyori  
          47.
          ParentQueue does not validate the queue on removal Sub-task Resolved Andras Gyori  
          48.
          Extend /scheduler endpoint with template properties Sub-task Resolved Andras Gyori  
          49.
          Allow definition of auto queue template properties in root Sub-task Resolved Andras Gyori  
          50.
          CS Flexible AQC: Add separate parent and leaf template property. Sub-task Resolved Andras Gyori  
          51.
          Fix Auto Queue template to properly set all configuration properties Sub-task Resolved Andras Gyori  
          52.
          Parents node labels are incorrectly added to child queues in weight mode Sub-task Resolved Benjamin Teke  

          Activity

            People

              Unassigned Unassigned
              wangda Wangda Tan
              Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 20m
                  4h 20m