Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496 [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
  3. YARN-10598

CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • capacity scheduler
    • None
    • Reviewed

    Description

      Under this umbrella (YARN-10496), weight-mode has been implemented for CS with YARN-10504.

      Auto-queue creation has been also implemented with YARN-10506.

      Connected to this effort, we would like to expose the type of the queue with the RM's /scheduler REST endpoint.

      To extend/modify the values added in YARN-10581 these 3 fields will describe a queue:

      • queueType : parent/leaf
      • creationMethod : static/dynamicLegacy/dynamicFlexible
      • autoCreationEligibility : off/legacy/flexible

      After this change here are some example cases:

      • Static parent queue which has the auto-creation-enabled-v2 false:
        • queueType : parent
        • creationMethod : static
        • autoCreationEligibility : off
      • Static managed parent (can have dynamic children):
        • queueType : parent
        • creationMethod : static
        • autoCreationEligibility : legacy
      • Legacy auto-created leaf queue (cannot have children):
        • queueType : leaf
        • creationMethod : dynamicLegacy
        • autoCreationEligibility : off
      • Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 true: 
        • queueType : parent
        • creationMethod : dynamicFlexible
        • autoCreationEligibility : flexible
      • Auto-created (v2) leaf queue (cannot have children):
        • queueType : leaf
        • creationMethod : dynamicFlexible
        • autoCreationEligibility : off

      Attachments

        1. YARN-10598.001.patch
          25 kB
          Benjamin Teke
        2. YARN-10598.002.patch
          23 kB
          Benjamin Teke
        3. YARN-10598.003.patch
          24 kB
          Benjamin Teke

        Issue Links

          Activity

            People

              bteke Benjamin Teke
              bteke Benjamin Teke
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: