[YARN-9055] Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.0
• Component/s: capacityscheduler
• Assignee: Aihua Xu
• Reporter: Aihua Xu
• Created: 26/Nov/18 18:30
• Updated: 13/Dec/18 02:04

Description

YARN-1582 adds the support of maximum-allocation-mb configuration per queue. That feature gives the flexibility to give different memory requirements for different queues. Such patch adds the limitation that the queue level configuration can't exceed the cluster level default configuration, but I feel it may make more sense to remove such limitation to allow any overrides since

1. Such configuration is controlled by the admin so it shouldn't get abused;
2. It's common that typical queues require standard size containers while some job (queues) have requirements for larger containers. With current limitation, we have to set larger configuration on the cluster setting which will cause resource abuse unless we override them on all the queues.

We can remove such limitation in CapacitySchedulerConfiguration.java so the cluster setting provides the default value and queue setting can override it.

   if (maxAllocationMbPerQueue > clusterMax.getMemorySize()
        || maxAllocationVcoresPerQueue > clusterMax.getVirtualCores()) {
      throw new IllegalArgumentException(
          "Queue maximum allocation cannot be larger than the cluster setting"
          + " for queue " + queue
          + " max allocation per queue: " + result
          + " cluster setting: " + clusterMax);
    }

Let me know if it makes sense.