Hadoop YARN
  1. Hadoop YARN
  2. YARN-343

Capacity Scheduler maximum-capacity value -1 is invalid

    Details

      Description

      I tried to start the resource manager using the capacity scheduler with a particular queues maximum-capacity set to -1 which is supposed to disable it according to the docs but I got the following exception:

      java.lang.IllegalArgumentException: Illegal value of maximumCapacity -0.01 used in call to setMaxCapacity for queue foo
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.checkMaxCapacity(CSQueueUtils.java:31)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.setupQueueConfigs(LeafQueue.java:220)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.<init>(LeafQueue.java:191)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:310)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:325)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:232)
      at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:202)

      1. YARN-343.1.patch
        2 kB
        Xuan Gong
      2. Yarn-343.2.patch
        1 kB
        Xuan Gong
      3. YARN-343.3.patch
        3 kB
        Xuan Gong

        Activity

        Hide
        Xuan Gong added a comment -

        The number we set up for the queue at CapacityScheduler.xml is configured as %. So here, you set -1, after read to the code, it will become -0.01. If you do not set any value at this property, it will use 100 as default, that means this queue will be fully used. And I do not think we can use negative number, it will check the value, and will pop up the exception if we set any negative number.

        Show
        Xuan Gong added a comment - The number we set up for the queue at CapacityScheduler.xml is configured as %. So here, you set -1, after read to the code, it will become -0.01. If you do not set any value at this property, it will use 100 as default, that means this queue will be fully used. And I do not think we can use negative number, it will check the value, and will pop up the exception if we set any negative number.
        Hide
        Thomas Graves added a comment -

        -1 was the value used in 1.X to also mean its disabled so we should support for backwards compatibility. Its also documented in the 0.23/2.0 capacity scheduler doc: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html

        Show
        Thomas Graves added a comment - -1 was the value used in 1.X to also mean its disabled so we should support for backwards compatibility. Its also documented in the 0.23/2.0 capacity scheduler doc: http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
        Hide
        Arun C Murthy added a comment -

        Agreed, looks like a bug.

        Show
        Arun C Murthy added a comment - Agreed, looks like a bug.
        Hide
        Xuan Gong added a comment -

        Set the default value as -1, and check if this default value is used. If yes, we will disable the property, aka, use 100%

        Show
        Xuan Gong added a comment - Set the default value as -1, and check if this default value is used. If yes, we will disable the property, aka, use 100%
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12565659/YARN-343.1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/368//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/368//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/368//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12565659/YARN-343.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/368//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/368//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/368//console This message is automatically generated.
        Hide
        Xuan Gong added a comment -

        should set default value as -1

        Show
        Xuan Gong added a comment - should set default value as -1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566567/Yarn-343.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/370//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/370//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566567/Yarn-343.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/370//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/370//console This message is automatically generated.
        Hide
        Xuan Gong added a comment -

        Add testcase to test maximum capacity setup

        Show
        Xuan Gong added a comment - Add testcase to test maximum capacity setup
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12566796/YARN-343.3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/372//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/372//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12566796/YARN-343.3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/372//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/372//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        +1 looks good. Thanks Xuan.

        Show
        Thomas Graves added a comment - +1 looks good. Thanks Xuan.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3300 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3300/)
        YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3300 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3300/ ) YARN-343 . Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Hide
        Arun C Murthy added a comment -

        Sorry, one thing I failed to notice...

        We really should return Float.MAX_VALUE rather than 100.0 since 100.0 will limit it to parent's max-cap, where-as Float.MAX_VALUE will let it go beyond. Does that make sense?

        Show
        Arun C Murthy added a comment - Sorry, one thing I failed to notice... We really should return Float.MAX_VALUE rather than 100.0 since 100.0 will limit it to parent's max-cap, where-as Float.MAX_VALUE will let it go beyond. Does that make sense?
        Hide
        Thomas Graves added a comment -


        I expect the parents max capacity to apply to its children so no child can get over the max capacity of the parent. Otherwise the parent max capacity has no meaning.

        Show
        Thomas Graves added a comment - I expect the parents max capacity to apply to its children so no child can get over the max capacity of the parent. Otherwise the parent max capacity has no meaning.
        Hide
        Thomas Graves added a comment -

        Note I think we discussed this under MAPREDUCE-3683.

        Show
        Thomas Graves added a comment - Note I think we discussed this under MAPREDUCE-3683 .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #113 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/113/)
        YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #113 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/113/ ) YARN-343 . Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #511 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/511/)
        YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440459)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440459
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #511 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/511/ ) YARN-343 . Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440459) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440459 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1302 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1302/)
        YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1302 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1302/ ) YARN-343 . Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1330 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1330/)
        YARN-343. Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457)

        Result = SUCCESS
        tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457
        Files :

        • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1330 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1330/ ) YARN-343 . Capacity Scheduler maximum-capacity value -1 is invalid (Xuan Gong via tgraves) (Revision 1440457) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1440457 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java

          People

          • Assignee:
            Xuan Gong
            Reporter:
            Thomas Graves
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development