[YARN-5545] Fix issues related to Max App in capacity scheduler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.9.0, 3.0.0-alpha2
• Component/s: capacity scheduler
• Labels: oct16-medium
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 21/Aug/16 12:57
• Updated: 11/Nov/16 19:31
• Resolved: 11/Nov/16 19:31

Description

Issues as part of Max apps in Capacity scheduler:
1. Cap total applications across the queue hierarchy based on existing max app calculation
2. Introduce a new configuration to take default max apps per queue irrespective of the queue capacity configuration
3. When the capacity configuration of the default partition is ZERO but queue has capacity for other partition then app is not getting submitted, though app is submitted in other partition

Steps to reproduce Issue 3 :

Configure capacity scheduler
yarn.scheduler.capacity.root.default.capacity=0
yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50

Submit application as below

./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar sleep -Dmapreduce.job.node-label-expression=labelx -Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1

2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
...
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001
	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
	... 25 more