XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 3.3.0
    • Component/s: capacity scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      After this change, capacity scheduler queue is able to inherit and override max-allocation from its parent queue. The subqueue's max-allocation can be larger or smaller than the parent queue's but can't exceed the global "yarn.scheduler.capacity.maximum-allocation". User can set queue-level max-allocation on any level of queues, using configuration property "yarn.scheduler.capacity.%QUEUE_PATH%.maximum-allocation". And this property allows user to set max-resource-allocation for customized resource types, e.g "memory=20G,vcores=20,gpu=3".
      Show
      After this change, capacity scheduler queue is able to inherit and override max-allocation from its parent queue. The subqueue's max-allocation can be larger or smaller than the parent queue's but can't exceed the global "yarn.scheduler.capacity.maximum-allocation". User can set queue-level max-allocation on any level of queues, using configuration property "yarn.scheduler.capacity.%QUEUE_PATH%.maximum-allocation". And this property allows user to set max-resource-allocation for customized resource types, e.g "memory=20G,vcores=20,gpu=3".

      Description

      YARN-1582 adds the support of maximum-allocation-mb configuration per queue which is targeting to support larger container features on dedicated queues (larger maximum-allocation-mb/maximum-allocation-vcores for such queue) . While to achieve larger container configuration, we need to increase the global maximum-allocation-mb/maximum-allocation-vcores (e.g. 120G/256) and then override those configurations with desired values on the queues since queue configuration can't be larger than cluster configuration. There are many queues in the system and if we forget to configure such values when adding a new queue, then such queue gets default 120G/256 which typically is not what we want.

      We can come up with a queue-default configuration (set to normal queue configuration like 16G/8), so the leaf queues gets such values by default.

        Attachments

        1. YARN-9116.1.patch
          25 kB
          Aihua Xu
        2. YARN-9116.2.patch
          56 kB
          Aihua Xu
        3. YARN-9116.3.patch
          52 kB
          Aihua Xu
        4. YARN-9116.4.patch
          55 kB
          Aihua Xu
        5. YARN-9116.5.patch
          55 kB
          Aihua Xu

          Activity

            People

            • Assignee:
              aihuaxu Aihua Xu
              Reporter:
              aihuaxu Aihua Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: