[YARN-8720] CapacityScheduler does not enforce max resource allocation check at queue level

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2, 2.8.6
• Component/s: capacity scheduler, capacityscheduler, resourcemanager
• Hadoop Flags: Reviewed
• Assignee: Tarun Parimi
• Reporter: Tarun Parimi
• Created: 27/Aug/18 14:17
• Updated: 10/Oct/18 02:22
• Resolved: 14/Sep/18 12:14

Description

The value of yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb/vcores is not strictly enforced when applications request containers. An InvalidResourceRequestException is thrown only when the ResourceRequest is greater than the global value of yarn.scheduler.maximum-allocation-mb/vcores . So for an example configuration such as below,

 

yarn.scheduler.maximum-allocation-mb=4096
yarn.scheduler.capacity.root.test.maximum-allocation-mb=2048

 

The below DSShell command runs successfully and asks an AM container of size 4096MB which is greater than max 2048MB configured in test queue.

yarn jar $YARN_HOME/hadoop-yarn-applications-distributedshell.jar -num_containers 1 -jar $YARN_HOME/hadoop-yarn-applications-distributedshell.jar -shell_command "sleep 60" -container_memory=4096 -master_memory=4096 -queue=test

Instead it should not launch the application and fail with InvalidResourceRequestException . The child container however will be requested with size 2048MB as DSShell AppMaster does the below check before ResourceRequest ask with RM.

// A resource ask cannot exceed the max.
if (containerMemory > maxMem) {
 LOG.info("Container memory specified above max threshold of cluster."
 + " Using max value." + ", specified=" + containerMemory + ", max="
 + maxMem);
 containerMemory = maxMem;
}

 

Issue Links

• causes: YARN-8858 CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.