[YARN-10869] CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.1
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: capacity scheduler
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Benjamin Teke
• Created: 21/Jul/21 16:51
• Updated: 16/Feb/22 13:48
• Resolved: 29/Jul/21 15:57

Description

When using auto created queues even though the default maximum allocation was overridden in yarn-site.xml, CS will throw the following exception if a dynamic queue has the maximum allocation set via templates (yarn.scheduler.capacity.root.users.leaf-queue-template.maximum-allocation-mb) above the default 8 GB memory/4 cores:

java.lang.IllegalArgumentException: Queue maximum allocation cannot be larger than the cluster setting for queue root.users.root max allocation per queue: <memory:10000, vCores:4> cluster setting: <memory:8192, vCores:4>
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupMaximumAllocation(AbstractCSQueue.java:550)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:413)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:186)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.<init>(LeafQueue.java:175)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.<init>(LeafQueue.java:156)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractAutoCreatedLeafQueue.<init>(AbstractAutoCreatedLeafQueue.java:54)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AutoCreatedLeafQueue.<init>(AutoCreatedLeafQueue.java:45)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createLegacyAutoQueue(CapacitySchedulerQueueManager.java:669)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createQueue(CapacitySchedulerQueueManager.java:541)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getOrCreateQueueFromPlacementContext(CapacityScheduler.java:969)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplication(CapacityScheduler.java:1029)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1989)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:171)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:1139)
resourcemanager      | 	at org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl$RMAppRecoveredTransition.transition(RMAppImpl.java:1090)

The reason for this is the following:

In ManagedParent#getLeafQueueConfigs a completely new CapacitySchedulerConfiguration gets created:

public CapacitySchedulerConfiguration getLeafQueueConfigs(
      CapacitySchedulerConfiguration templateConfig, String leafQueueName) {
    CapacitySchedulerConfiguration leafQueueConfigTemplate = new
        CapacitySchedulerConfiguration(new Configuration(false), false);
    for (final Iterator<Map.Entry<String, String>> iterator =
         templateConfig.iterator(); iterator.hasNext(); ) {
      Map.Entry<String, String> confKeyValuePair = iterator.next();
      final String name = confKeyValuePair.getKey().replaceFirst(
          CapacitySchedulerConfiguration
              .AUTO_CREATED_LEAF_QUEUE_TEMPLATE_PREFIX,
          leafQueueName);
      leafQueueConfigTemplate.set(name, confKeyValuePair.getValue());
    }
    return leafQueueConfigTemplate;
  }
}

This only contains the template configs related to the auto created queue, copied from the original Configuration object (and loaded from capacity-scheduler.xml). The maximum-allocation calculation was refactored in YARN-9116:

private void setupMaximumAllocation(CapacitySchedulerConfiguration csConf) {
    String myQueuePath = getQueuePath();
    Resource clusterMax = ResourceUtils
        .fetchMaximumAllocationFromConfig(csConf);
    Resource queueMax = csConf.getQueueMaximumAllocation(myQueuePath);

    maximumAllocation = Resources.clone(
        parent == null ? clusterMax : parent.getMaximumAllocation());

    String errMsg =
        "Queue maximum allocation cannot be larger than the cluster setting"
            + " for queue " + myQueuePath
            + " max allocation per queue: %s"
            + " cluster setting: " + clusterMax;

    if (queueMax == Resources.none()) {
      // Handle backward compatibility
      long queueMemory = csConf.getQueueMaximumAllocationMb(myQueuePath);
      int queueVcores = csConf.getQueueMaximumAllocationVcores(myQueuePath);
      if (queueMemory != UNDEFINED) {
        maximumAllocation.setMemorySize(queueMemory);
      }

      if (queueVcores != UNDEFINED) {
        maximumAllocation.setVirtualCores(queueVcores);
      }

      if ((queueMemory != UNDEFINED && queueMemory > clusterMax.getMemorySize()
          || (queueVcores != UNDEFINED
          && queueVcores > clusterMax.getVirtualCores()))) {
        throw new IllegalArgumentException(
            String.format(errMsg, maximumAllocation));
      }
    } else {
      // Queue level maximum-allocation can't be larger than cluster setting
      for (ResourceInformation ri : queueMax.getResources()) {
        if (ri.compareTo(clusterMax.getResourceInformation(ri.getName())) > 0) {
          throw new IllegalArgumentException(String.format(errMsg, queueMax));
        }

        maximumAllocation.setResourceInformation(ri.getName(), ri);
      }
    }
  }

Let's consider the following scenarios:

1. No maximum-allocation is set through templates, neither through the old maximum-allocation-mb/vcore property: queueMax will get the value Resources.none(), so its if condition evaluates to true but both queueMemory and queueVcores will be UNDEFINED. The maximumAllocation will simply be inherited from the parent and no clusterMax comparison will be done (the second if will be skipped).
2. One of the maximum-allocation-mb/vcore properties is set: a comparison will be executed to check whether the value is indeed lower than the cluster-wide maximum. Here comes the getLeafQueueConfigs' CapacitySchedulerConfiguration duplication into the picture. Since the cluster-wide maximum is a property that comes from the YarnConfiguration object and the copied config object gets a newly created Configuration object it'll only contain the default properties.

There are multiple solutions to this problem: either the cluster-wide maximum allocation should be migrated to the cloned Configuration object or when checking the maximum allocation the original Configuration object should be used.

YARN-9569 solved this issue partially, but the old yarn.scheduler.maximum-allocation-mb/vcore is not migrated.

Issue Links

• relates to: YARN-9569 Auto-created leaf queues do not honor cluster-wide min/max memory/vcores
• links to: GitHub Pull Request #3225
• links to: GitHub Pull Request #3504