Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6024

Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Found one corner case when continuous reservation looking doesn't work:

      When queue's used=max, the queue's capacity check fails.

      1. YARN-6024.001.patch
        3 kB
        Wangda Tan
      2. YARN-6024-branch-2.7.001.patch
        7 kB
        Wangda Tan
      3. YARN-6024-branch-2.7.001.patch
        7 kB
        Wangda Tan

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        The issue exists in branch-2.7, and not existed in branch-2.8/branch-2/trunk.

        The root cause is, in AbstractCSQueue#canAssignToThisQueue:

                  if (Resources.lessThan(resourceCalculator, clusterResource,
                      newTotalWithoutReservedResource, currentLimitResource)) {
                    if (LOG.isDebugEnabled()) {
                      LOG.debug("try to use reserved: " + getQueueName()
                          + " usedResources: " + queueUsage.getUsed()
                          + ", clusterResources: " + clusterResource
                          + ", reservedResources: " + resourceCouldBeUnreserved
                          + ", capacity-without-reserved: "
                          + newTotalWithoutReservedResource + ", maxLimitCapacity: "
                          + currentLimitResource);
                    }
                    currentResourceLimits.setAmountNeededUnreserve(Resources.subtract(newTotalResource,
                        currentLimitResource));
                    return true;
                  }
        

        It should be lessThanOrEqual instead of lessThan.

        But we need to backport test cases to branch-2.8/branch-2/trunk to prevent future changes.

        Show
        leftnoteasy Wangda Tan added a comment - The issue exists in branch-2.7, and not existed in branch-2.8/branch-2/trunk. The root cause is, in AbstractCSQueue#canAssignToThisQueue: if (Resources.lessThan(resourceCalculator, clusterResource, newTotalWithoutReservedResource, currentLimitResource)) { if (LOG.isDebugEnabled()) { LOG.debug( " try to use reserved: " + getQueueName() + " usedResources: " + queueUsage.getUsed() + ", clusterResources: " + clusterResource + ", reservedResources: " + resourceCouldBeUnreserved + ", capacity-without-reserved: " + newTotalWithoutReservedResource + ", maxLimitCapacity: " + currentLimitResource); } currentResourceLimits.setAmountNeededUnreserve(Resources.subtract(newTotalResource, currentLimitResource)); return true ; } It should be lessThanOrEqual instead of lessThan . But we need to backport test cases to branch-2.8/branch-2/trunk to prevent future changes.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached fix for branch-2.7.

        Show
        leftnoteasy Wangda Tan added a comment - Attached fix for branch-2.7.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached unit test changes to trunk to avoid future regression.

        Show
        leftnoteasy Wangda Tan added a comment - Attached unit test changes to trunk to avoid future regression.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G could you help review the change?

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G could you help review the change?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 11m 45s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        -1 mvninstall 0m 11s root in trunk failed.
        +1 compile 1m 19s trunk passed with JDK v1.8.0_111
        -1 compile 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.7.0_121.
        +1 checkstyle 0m 42s trunk passed
        -1 mvnsite 0m 30s hadoop-yarn-server-resourcemanager in trunk failed.
        -1 mvneclipse 0m 25s hadoop-yarn-server-resourcemanager in trunk failed.
        -1 findbugs 0m 24s hadoop-yarn-server-resourcemanager in trunk failed.
        +1 javadoc 0m 55s trunk passed with JDK v1.8.0_111
        -1 javadoc 0m 28s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.7.0_121.
        -1 mvninstall 0m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 compile 1m 25s the patch passed with JDK v1.8.0_111
        +1 javac 1m 25s the patch passed
        -1 compile 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        -1 javac 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13)
        -1 mvnsite 0m 29s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 mvneclipse 0m 26s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_111
        -1 javadoc 0m 27s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        -1 unit 0m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        66m 25s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue YARN-6024
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844470/YARN-6024.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux fb742ca5c834 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22befbd
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvneclipse https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvneclipse-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvneclipse https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvneclipse-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14444/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14444/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 45s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. -1 mvninstall 0m 11s root in trunk failed. +1 compile 1m 19s trunk passed with JDK v1.8.0_111 -1 compile 0m 25s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.7.0_121. +1 checkstyle 0m 42s trunk passed -1 mvnsite 0m 30s hadoop-yarn-server-resourcemanager in trunk failed. -1 mvneclipse 0m 25s hadoop-yarn-server-resourcemanager in trunk failed. -1 findbugs 0m 24s hadoop-yarn-server-resourcemanager in trunk failed. +1 javadoc 0m 55s trunk passed with JDK v1.8.0_111 -1 javadoc 0m 28s hadoop-yarn-server-resourcemanager in trunk failed with JDK v1.7.0_121. -1 mvninstall 0m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 compile 1m 25s the patch passed with JDK v1.8.0_111 +1 javac 1m 25s the patch passed -1 compile 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. -1 javac 0m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) -1 mvnsite 0m 29s hadoop-yarn-server-resourcemanager in the patch failed. -1 mvneclipse 0m 26s hadoop-yarn-server-resourcemanager in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 56s the patch passed with JDK v1.8.0_111 -1 javadoc 0m 27s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. -1 unit 0m 9s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 66m 25s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue YARN-6024 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844470/YARN-6024.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fb742ca5c834 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22befbd Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvneclipse https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-mvneclipse-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvneclipse https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-mvneclipse-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14444/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14444/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14444/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 50s trunk passed
        +1 compile 0m 40s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 42s trunk passed
        +1 mvneclipse 0m 21s trunk passed
        +1 findbugs 1m 12s trunk passed
        +1 javadoc 0m 24s trunk passed
        +1 mvninstall 0m 36s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13)
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 21s the patch passed
        +1 javadoc 0m 22s the patch passed
        +1 unit 38m 35s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        61m 51s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6024
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844470/YARN-6024.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux be9d554e87ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 22befbd
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14445/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14445/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14445/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 50s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 42s trunk passed +1 mvneclipse 0m 21s trunk passed +1 findbugs 1m 12s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed -0 checkstyle 0m 19s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 13 unchanged - 0 fixed = 14 total (was 13) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 21s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 38m 35s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 61m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6024 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844470/YARN-6024.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux be9d554e87ee 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 22befbd Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14445/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14445/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14445/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Reattached branch-2.7 to trigger Jenkins.

        Show
        leftnoteasy Wangda Tan added a comment - Reattached branch-2.7 to trigger Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
        +1 mvninstall 7m 43s branch-2.7 passed
        +1 compile 0m 23s branch-2.7 passed with JDK v1.8.0_111
        +1 compile 0m 28s branch-2.7 passed with JDK v1.7.0_121
        +1 checkstyle 0m 20s branch-2.7 passed
        +1 mvnsite 0m 35s branch-2.7 passed
        +1 mvneclipse 0m 16s branch-2.7 passed
        +1 findbugs 1m 4s branch-2.7 passed
        +1 javadoc 0m 19s branch-2.7 passed with JDK v1.8.0_111
        +1 javadoc 0m 23s branch-2.7 passed with JDK v1.7.0_121
        +1 mvninstall 0m 27s the patch passed
        +1 compile 0m 22s the patch passed with JDK v1.8.0_111
        +1 javac 0m 22s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_121
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        -1 whitespace 0m 1s The patch has 1771 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
        -1 whitespace 0m 49s The patch 70 line(s) with tabs.
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 0m 16s the patch passed with JDK v1.8.0_111
        +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121
        -1 unit 50m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        118m 21s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization
          hadoop.yarn.server.resourcemanager.TestClientRMTokens



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c420dfe
        JIRA Issue YARN-6024
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844479/YARN-6024-branch-2.7.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 782757b7ae6d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / 3e0bd99
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/whitespace-tabs.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14446/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14446/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. +1 mvninstall 7m 43s branch-2.7 passed +1 compile 0m 23s branch-2.7 passed with JDK v1.8.0_111 +1 compile 0m 28s branch-2.7 passed with JDK v1.7.0_121 +1 checkstyle 0m 20s branch-2.7 passed +1 mvnsite 0m 35s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed +1 findbugs 1m 4s branch-2.7 passed +1 javadoc 0m 19s branch-2.7 passed with JDK v1.8.0_111 +1 javadoc 0m 23s branch-2.7 passed with JDK v1.7.0_121 +1 mvninstall 0m 27s the patch passed +1 compile 0m 22s the patch passed with JDK v1.8.0_111 +1 javac 0m 22s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_121 +1 javac 0m 25s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 1s The patch has 1771 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply -1 whitespace 0m 49s The patch 70 line(s) with tabs. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 16s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 21s the patch passed with JDK v1.7.0_121 -1 unit 50m 28s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 118m 21s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestAMAuthorization   hadoop.yarn.server.resourcemanager.TestClientRMTokens Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Issue YARN-6024 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844479/YARN-6024-branch-2.7.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 782757b7ae6d 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 3e0bd99 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/whitespace-tabs.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14446/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14446/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14446/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Ying Zhang Ying Zhang added a comment -

        Wangda Tan, why we don't get rid of the "nowRequired", calculate nowTotalUsed instead of "newTotalResource", to make it exactly the same behavior as in 2.8? Just curious why, I'm fine in either way

        Show
        Ying Zhang Ying Zhang added a comment - Wangda Tan , why we don't get rid of the "nowRequired", calculate nowTotalUsed instead of "newTotalResource", to make it exactly the same behavior as in 2.8? Just curious why, I'm fine in either way
        Hide
        sunilg Sunil G added a comment -

        I am generally fine with approach taken for 2.7 patch.
        One comment getConfigurationWithMultipleQueues, could we pass capacity as arguments so that we can make it more generalized.

        I have not really understood the comment mentioned by Ying Zhang above.

        Show
        sunilg Sunil G added a comment - I am generally fine with approach taken for 2.7 patch. One comment getConfigurationWithMultipleQueues , could we pass capacity as arguments so that we can make it more generalized. I have not really understood the comment mentioned by Ying Zhang above.
        Hide
        sunilg Sunil G added a comment -

        +1 for branch-2.7 patch. If others does not have any difference of opinion, I will commit core patch in 2.7 and test case in trunk/branch-2.

        Show
        sunilg Sunil G added a comment - +1 for branch-2.7 patch. If others does not have any difference of opinion, I will commit core patch in 2.7 and test case in trunk/branch-2.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for review, Sunil G / Ying Zhang.

        Ying Zhang, I may not fully understand what you suggested, if it is just a code style change, I would prefer not to do it, since branch-2.7 is under maintenance state, keep changes to branch-2.7 simple and straightforward is more important. Make sense?

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for review, Sunil G / Ying Zhang . Ying Zhang , I may not fully understand what you suggested, if it is just a code style change, I would prefer not to do it, since branch-2.7 is under maintenance state, keep changes to branch-2.7 simple and straightforward is more important. Make sense?
        Hide
        Ying Zhang Ying Zhang added a comment - - edited

        Sorry for the confusion, Wangda Tan and Sunil G
        Yes, it totally makes sense to me, please go ahead.

        Here is what I mean anyway:
        In 2.7.3, we are comparing:

        newTotalWithoutReservedResource (which is like "totalUsed + newly_required - resourceCouldBeUnreserved") <= currentLimitResource

        With this, we are only checking the queue when "queue's available resource >= newly_required".
        In 2.8, we are comparing:

        newTotalWithoutReservedResource (which is like "totalUsed - resourceCouldBeUnreserved") < currentLimitResource

        With this, we are checking the queue as long as "queue's available resource > 0".
        There is a slightly difference.

        Show
        Ying Zhang Ying Zhang added a comment - - edited Sorry for the confusion, Wangda Tan and Sunil G Yes, it totally makes sense to me, please go ahead. Here is what I mean anyway: In 2.7.3, we are comparing: newTotalWithoutReservedResource (which is like "totalUsed + newly_required - resourceCouldBeUnreserved" ) <= currentLimitResource With this, we are only checking the queue when "queue's available resource >= newly_required". In 2.8, we are comparing: newTotalWithoutReservedResource (which is like "totalUsed - resourceCouldBeUnreserved" ) < currentLimitResource With this, we are checking the queue as long as "queue's available resource > 0". There is a slightly difference.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Ying Zhang,

        Gotcha, thanks for elaborate. Yeah it was a historical issue, I remember there're some changes after branch-2.8 to get rid of newRequired, but there're also some changes in application / leaf queue to handle resource-limit and headroom properly (Like respect parent queue's max capacity, etc.). I cannot remember all details, but to avoid regression, let's just focus changes in the patch.

        Show
        leftnoteasy Wangda Tan added a comment - Ying Zhang , Gotcha, thanks for elaborate. Yeah it was a historical issue, I remember there're some changes after branch-2.8 to get rid of newRequired, but there're also some changes in application / leaf queue to handle resource-limit and headroom properly (Like respect parent queue's max capacity, etc.). I cannot remember all details, but to avoid regression, let's just focus changes in the patch.
        Hide
        Ying Zhang Ying Zhang added a comment -

        Thanks for the details

        Show
        Ying Zhang Ying Zhang added a comment - Thanks for the details
        Hide
        sunilg Sunil G added a comment -

        +1 Committing.

        Show
        sunilg Sunil G added a comment - +1 Committing.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11048 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11048/)
        YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't (sunilg: rev f6715b26b65fc127b35368d0460b786ece88b5b8)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11048 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11048/ ) YARN-6024 . Capacity Scheduler 'continuous reservation looking' doesn't (sunilg: rev f6715b26b65fc127b35368d0460b786ece88b5b8) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java
        Hide
        sunilg Sunil G added a comment -

        Committed YARN-6024.001.patch to trunk/branch-2/branch-2.8 and YARN-6024-branch-2.7.001.patch to branch-2.7. Thanks Wangda Tan for the patch and thanks Ying Zhang for additional review.

        Show
        sunilg Sunil G added a comment - Committed YARN-6024 .001.patch to trunk/branch-2/branch-2.8 and YARN-6024 -branch-2.7.001.patch to branch-2.7. Thanks Wangda Tan for the patch and thanks Ying Zhang for additional review.

          People

          • Assignee:
            leftnoteasy Wangda Tan
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development