Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5453

FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource

    Details

    • Hadoop Flags:
      Reviewed

      Description

            demand = Resources.createResource(0);
            for (FSQueue childQueue : childQueues) {
              childQueue.updateDemand();
              Resource toAdd = childQueue.getDemand();
              demand = Resources.add(demand, toAdd);
              demand = Resources.componentwiseMin(demand, maxRes);
              if (Resources.equals(demand, maxRes)) {
                break;
              }
            }
      

      if one singe queue's demand resource exceed maxRes, the other queue's demand resource will not update.

      1. YARN-5453.01.patch
        4 kB
        sandflee
      2. YARN-5453.02.patch
        19 kB
        sandflee
      3. YARN-5453.03.patch
        3 kB
        sandflee
      4. YARN-5453.04.patch
        3 kB
        sandflee
      5. YARN-5453.05.patch
        6 kB
        sandflee

        Activity

        Hide
        sandflee sandflee added a comment -

        thanks Karthik Kambatla for review and commit !

        Show
        sandflee sandflee added a comment - thanks Karthik Kambatla for review and commit !
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10811/)
        YARN-5453. FairScheduler#update may skip update demand resource of child (kasha: rev 86ac1ad9fd65c7dd12278372b369de38dc4616db)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10811 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10811/ ) YARN-5453 . FairScheduler#update may skip update demand resource of child (kasha: rev 86ac1ad9fd65c7dd12278372b369de38dc4616db) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSParentQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
        Hide
        kasha Karthik Kambatla added a comment -

        Thanks sandflee for the contribution. Just committed this to trunk and branch-2.

        Show
        kasha Karthik Kambatla added a comment - Thanks sandflee for the contribution. Just committed this to trunk and branch-2.
        Hide
        kasha Karthik Kambatla added a comment -

        +1. Checking this in..

        Show
        kasha Karthik Kambatla added a comment - +1. Checking this in..
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 28s trunk passed
        +1 compile 0m 39s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 45s trunk passed
        +1 mvneclipse 0m 20s trunk passed
        +1 findbugs 1m 7s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 38s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 9s the patch passed
        +1 javadoc 0m 24s the patch passed
        +1 unit 39m 49s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        57m 37s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue YARN-5453
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837687/YARN-5453.05.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux c660c8d72058 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ca33bdd
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13804/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13804/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 28s trunk passed +1 compile 0m 39s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 7s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 34s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 9s the patch passed +1 javadoc 0m 24s the patch passed +1 unit 39m 49s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 57m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-5453 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837687/YARN-5453.05.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c660c8d72058 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ca33bdd Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13804/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13804/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sandflee sandflee added a comment -

        thanks Karthik Kambatla, patch updated.

        Show
        sandflee sandflee added a comment - thanks Karthik Kambatla , patch updated.
        Hide
        kasha Karthik Kambatla added a comment -

        The patch looks good. Comments:

        1. Nit: Let us add a comment as below when we limit demand to maxShare:
                // Cap demand to maxShare to limit allocation to maxShare
                demand = Resources.componentwiseMin(demand, maxShare);
          
        2. Can we add a unit test to ensure we don't regress in the future?
        Show
        kasha Karthik Kambatla added a comment - The patch looks good. Comments: Nit: Let us add a comment as below when we limit demand to maxShare: // Cap demand to maxShare to limit allocation to maxShare demand = Resources.componentwiseMin(demand, maxShare); Can we add a unit test to ensure we don't regress in the future?
        Hide
        sandflee sandflee added a comment -

        Hi, Karthik Kambatla, patch updated, failed test could run pass locally seems not related.

        Show
        sandflee sandflee added a comment - Hi, Karthik Kambatla , patch updated, failed test could run pass locally seems not related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 12s trunk passed
        +1 compile 0m 37s trunk passed
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 0m 42s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 11s trunk passed
        +1 javadoc 0m 25s trunk passed
        +1 mvninstall 0m 40s the patch passed
        +1 compile 0m 38s the patch passed
        +1 javac 0m 38s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 40s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 13s the patch passed
        +1 javadoc 0m 22s the patch passed
        -1 unit 36m 46s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        54m 0s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMHA
          hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831871/YARN-5453.04.patch
        JIRA Issue YARN-5453
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d8988084463a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 272a217
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13304/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13304/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13304/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13304/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 12s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 24s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 11s trunk passed +1 javadoc 0m 25s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 0m 38s the patch passed +1 javac 0m 38s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 13s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 36m 46s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 54m 0s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMHA   hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831871/YARN-5453.04.patch JIRA Issue YARN-5453 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d8988084463a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 272a217 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/13304/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13304/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13304/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13304/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Sorry for the delay in getting around to this. The patch does not apply any more. sandflee - mind updating the patch? I ll try to get to it sooner.

        Show
        kasha Karthik Kambatla added a comment - Sorry for the delay in getting around to this. The patch does not apply any more. sandflee - mind updating the patch? I ll try to get to it sooner.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 50s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 0m 57s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 31s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        +1 checkstyle 0m 17s the patch passed
        +1 mvnsite 0m 36s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 6s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 37m 30s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        52m 9s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822933/YARN-5453.03.patch
        JIRA Issue YARN-5453
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f500dadb0973 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d00d3ad
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12709/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12709/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 50s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 57s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 31s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 36s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 6s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 37m 30s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 52m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822933/YARN-5453.03.patch JIRA Issue YARN-5453 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f500dadb0973 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d00d3ad Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12709/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12709/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 32s trunk passed
        +1 compile 0m 35s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 40s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 4s trunk passed
        +1 javadoc 0m 22s trunk passed
        -1 mvninstall 0m 27s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 compile 0m 34s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 javac 0m 34s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 5 unchanged - 2 fixed = 8 total (was 7)
        -1 mvnsite 0m 20s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 mvneclipse 0m 16s the patch passed
        -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 findbugs 0m 17s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 0m 19s the patch passed
        -1 unit 0m 19s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        14m 46s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822913/YARN-5453.02.patch
        JIRA Issue YARN-5453
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 4c4e3243fe1a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 9c6a438
        Default Java 1.8.0_101
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12707/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12707/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 32s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 40s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 4s trunk passed +1 javadoc 0m 22s trunk passed -1 mvninstall 0m 27s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 0m 34s hadoop-yarn-server-resourcemanager in the patch failed. -1 javac 0m 34s hadoop-yarn-server-resourcemanager in the patch failed. -1 checkstyle 0m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 5 unchanged - 2 fixed = 8 total (was 7) -1 mvnsite 0m 20s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 16s the patch passed -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 17s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 19s the patch passed -1 unit 0m 19s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 14m 46s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822913/YARN-5453.02.patch JIRA Issue YARN-5453 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4c4e3243fe1a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 9c6a438 Default Java 1.8.0_101 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12707/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12707/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12707/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        sandflee sandflee added a comment -

        agree, update the patch as you suggested.

        Show
        sandflee sandflee added a comment - agree, update the patch as you suggested.
        Hide
        kasha Karthik Kambatla added a comment -

        If we can limit the scheduling to under maxResources, I don't see a reason to limit the demand. Limiting to maxResources could be an optimization, but I don't think that really buys us much.

        Show
        kasha Karthik Kambatla added a comment - If we can limit the scheduling to under maxResources, I don't see a reason to limit the demand. Limiting to maxResources could be an optimization, but I don't think that really buys us much.
        Hide
        sandflee sandflee added a comment -

        thanks for your explain,

        the scheduler allows allocations over fairshare and hence over maxResources.

        scheduler should grant that allocations should not exceed maxResource, seems unnecessary to limit demand to maxResource, thought?

        Show
        sandflee sandflee added a comment - thanks for your explain, the scheduler allows allocations over fairshare and hence over maxResources. scheduler should grant that allocations should not exceed maxResource, seems unnecessary to limit demand to maxResource, thought?
        Hide
        kasha Karthik Kambatla added a comment -

        Demand is limited to maxResources to enforce maxResources: If we don't limit the demand, demand could be larger than maxResources. Even though the fairshare is limited by maxResources, the scheduler allows allocations over fairshare and hence over maxResources.

        We should document this in the code itself for posterity.

        In the snippet posted in the description, why not change break to continue and move demand = componentWiseMin(demand, maxRes); outside the for loop.

        Show
        kasha Karthik Kambatla added a comment - Demand is limited to maxResources to enforce maxResources: If we don't limit the demand, demand could be larger than maxResources. Even though the fairshare is limited by maxResources, the scheduler allows allocations over fairshare and hence over maxResources. We should document this in the code itself for posterity. In the snippet posted in the description, why not change break to continue and move demand = componentWiseMin(demand, maxRes); outside the for loop.
        Hide
        sandflee sandflee added a comment -

        any reason to limit resource demand to maxResource? cc Karthik Kambatla

        Show
        sandflee sandflee added a comment - any reason to limit resource demand to maxResource? cc Karthik Kambatla
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 54s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 20s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        +1 checkstyle 0m 18s the patch passed
        +1 mvnsite 0m 39s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 8s the patch passed
        +1 javadoc 0m 19s the patch passed
        -1 unit 38m 7s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        53m 11s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821858/YARN-5453.01.patch
        JIRA Issue YARN-5453
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux e57a2da6689e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2d82276
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12628/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/12628/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 54s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 20s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed +1 checkstyle 0m 18s the patch passed +1 mvnsite 0m 39s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 8s the patch passed +1 javadoc 0m 19s the patch passed -1 unit 38m 7s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 53m 11s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSLeafQueue Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821858/YARN-5453.01.patch JIRA Issue YARN-5453 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e57a2da6689e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d82276 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12628/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12628/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/12628/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.

          People

          • Assignee:
            sandflee sandflee
            Reporter:
            sandflee sandflee
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development