Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9569

Auto-created leaf queues do not honor cluster-wide min/max memory/vcores

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.3.0
    • Component/s: capacity scheduler
    • Labels:
      None

      Description

      Auto-created leaf queues do not honor cluster-wide settings for maximum CPU/vcores allocation.

      To reproduce:

      1. Set auto-create-child-queue.enabled=true for a parent queue.
      2. Set leaf-queue-template.maximum-allocation-mb=16384.
      3. Set yarn.resource-types.memory-mb.maximum-allocation=16384 in resource-types.xml
      4. Launch a YARN app with a container requesting 16 GB RAM.

       

      This scenario should work, but instead you get an error similar to this:

      java.lang.IllegalArgumentException: Queue maximum allocation cannot be larger than the cluster setting for queue root.auto.test max allocation per queue: <memory:16384, vCores:1> cluster setting: <memory:8192, vCores:4>   

       

      This seems to be caused by this code in ManagedParentQueue.getLeafQueueConfigs:

      CapacitySchedulerConfiguration leafQueueConfigTemplate = new
          CapacitySchedulerConfiguration(new Configuration(false), false);

       

      This initializes a new leaf queue configuration that does not read resource-types.xml (or any other config). Later, this CapacitySchedulerConfiguration instance calls ResourceUtils.fetchMaximumAllocationFromConfig()  from its getMaximumAllocationPerQueue() method and passes itself as the configuration to use. Since the resource types are not present, ResourceUtils falls back to compiled-in defaults of 8GB RAM, 4 cores.

       

      I was able to work around this with a custom AutoCreatedQueueManagementPolicy implementation which does something like this in init() and reinitialize():

      for (Map.Entry<String, String> entry : this.scheduler.getConfiguration()) {
      if (entry.getKey().startsWith("yarn.resource-types")) {
        parentQueue.getLeafQueueTemplate().getLeafQueueConfigs()
          .set(entry.getKey(), entry.getValue());
        }
      }
      

      However, this is obviously a very hacky way to solve the problem.

      I can submit a proper patch if someone can provide some direction as to the best way to proceed.

       

        Attachments

        1. YARN-9569.001.patch
          2 kB
          Craig Condit
        2. YARN-9569.002.patch
          9 kB
          Craig Condit

          Activity

            People

            • Assignee:
              ccondit Craig Condit
              Reporter:
              ccondit Craig Condit
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: