Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5545

Fix issues related to Max App in capacity scheduler

    Details

    • Hadoop Flags:
      Reviewed

      Description

      Issues as part of Max apps in Capacity scheduler:
      1. Cap total applications across the queue hierarchy based on existing max app calculation
      2. Introduce a new configuration to take default max apps per queue irrespective of the queue capacity configuration
      3. When the capacity configuration of the default partition is ZERO but queue has capacity for other partition then app is not getting submitted, though app is submitted in other partition

      Steps to reproduce Issue 3 :

      Configure capacity scheduler
      yarn.scheduler.capacity.root.default.capacity=0
      yarn.scheduler.capacity.root.queue1.accessible-node-labels.labelx.capacity=50
      yarn.scheduler.capacity.root.default.accessible-node-labels.labelx.capacity=50

      Submit application as below

      ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.0.0-alpha2-SNAPSHOT-tests.jar sleep -Dmapreduce.job.node-label-expression=labelx -Dmapreduce.job.queuename=default -m 1 -r 1 -mt 10000000 -rt 1

      2016-08-21 18:21:31,375 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1471670113386_0001
      java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:316)
      	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:255)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1344)
      	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
      ...
      Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1471670113386_0001 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 0 applications, cannot accept submission of application: application_1471670113386_0001
      	at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:286)
      	at org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:296)
      	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
      	... 25 more
      

        Attachments

        1. capacity-scheduler.xml
          4 kB
          Bibin A Chundatt
        2. YARN-5545.0001.patch
          21 kB
          Bibin A Chundatt
        3. YARN-5545.0002.patch
          22 kB
          Bibin A Chundatt
        4. YARN-5545.0003.patch
          22 kB
          Bibin A Chundatt
        5. YARN-5545.0005.patch
          14 kB
          Bibin A Chundatt
        6. YARN-5545.0006.patch
          14 kB
          Bibin A Chundatt
        7. YARN-5545.0007.patch
          15 kB
          Bibin A Chundatt
        8. YARN-5545.0008.patch
          15 kB
          Bibin A Chundatt
        9. YARN-5545.004.patch
          15 kB
          Bibin A Chundatt

          Activity

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              bibinchundatt Bibin A Chundatt
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: