Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6406

Remove SchedulerRequestKeys when no more pending ResourceRequest

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 2.7.3, 3.0.0-alpha2
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys from the AppScheduleingInfo. It looks like after YARN-6040, ScedulerRequestKeys are removed only if the Application sends a 0 numContainers requests. While earlier, the outstanding schedulerKeys were also remove as soon as a container is allocated as well.

      An additional optimization we were hoping to include is to remove the ResourceRequests itself once the numContainers == 0, since we see in our clusters that the RM heap space consumption increases drastically due to a large number of ResourceRequests with 0 num containers.

      1. YARN-6406.001.patch
        22 kB
        Arun Suresh
      2. YARN-6406.002.patch
        28 kB
        Arun Suresh
      3. YARN-6406-branch-2.001.patch
        29 kB
        Arun Suresh
      4. YARN-6406-branch-2.002.patch
        30 kB
        Arun Suresh
      5. YARN-6406-branch-2.003.patch
        30 kB
        Arun Suresh

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
        YARN-6406. Remove SchedulerRequestKeys when no more pending (wangda: rev 87e2ef8c985bb72a916477e8783359f2859f7890)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) YARN-6406 . Remove SchedulerRequestKeys when no more pending (wangda: rev 87e2ef8c985bb72a916477e8783359f2859f7890) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java
        Hide
        asuresh Arun Suresh added a comment -

        Cherry-picked to branch-2.

        Show
        asuresh Arun Suresh added a comment - Cherry-picked to branch-2.
        Hide
        asuresh Arun Suresh added a comment -

        Committing this shortly.
        The test failures are not related.

        Show
        asuresh Arun Suresh added a comment - Committing this shortly. The test failures are not related.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 21s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 7m 21s branch-2 passed
        +1 compile 0m 33s branch-2 passed with JDK v1.8.0_121
        +1 compile 0m 34s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 29s branch-2 passed
        +1 mvnsite 0m 39s branch-2 passed
        +1 mvneclipse 0m 18s branch-2 passed
        +1 findbugs 1m 14s branch-2 passed
        +1 javadoc 0m 23s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_121
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.8.0_121
        +1 javac 0m 28s the patch passed
        +1 compile 0m 30s the patch passed with JDK v1.7.0_121
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 589 unchanged - 7 fixed = 591 total (was 596)
        +1 mvnsite 0m 37s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 23s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_121
        +1 javadoc 0m 23s the patch passed with JDK v1.7.0_121
        -1 unit 40m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        99m 12s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:8515d35
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863575/YARN-6406-branch-2.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux cef4ec2e6e93 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / d4f553d
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15649/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15649/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15649/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 21s branch-2 passed +1 compile 0m 33s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 34s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 29s branch-2 passed +1 mvnsite 0m 39s branch-2 passed +1 mvneclipse 0m 18s branch-2 passed +1 findbugs 1m 14s branch-2 passed +1 javadoc 0m 23s branch-2 passed with JDK v1.8.0_121 +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 34s the patch passed +1 compile 0m 28s the patch passed with JDK v1.8.0_121 +1 javac 0m 28s the patch passed +1 compile 0m 30s the patch passed with JDK v1.7.0_121 +1 javac 0m 30s the patch passed -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 2 new + 589 unchanged - 7 fixed = 591 total (was 596) +1 mvnsite 0m 37s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 23s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 23s the patch passed with JDK v1.7.0_121 -1 unit 40m 1s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 99m 12s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:8515d35 JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863575/YARN-6406-branch-2.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cef4ec2e6e93 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d4f553d Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15649/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15649/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15649/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15649/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Fixing checkstyle issues. TestCapacityScheduler runs fine locally. Will cherry-pick this into branch-2 after a good Jenkins runs

        Show
        asuresh Arun Suresh added a comment - Fixing checkstyle issues. TestCapacityScheduler runs fine locally. Will cherry-pick this into branch-2 after a good Jenkins runs
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 14m 42s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 7m 25s branch-2 passed
        +1 compile 0m 35s branch-2 passed with JDK v1.8.0_121
        +1 compile 0m 38s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 36s branch-2 passed
        +1 mvnsite 0m 43s branch-2 passed
        +1 mvneclipse 0m 22s branch-2 passed
        +1 findbugs 1m 21s branch-2 passed
        +1 javadoc 0m 25s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 0m 29s branch-2 passed with JDK v1.7.0_121
        +1 mvninstall 0m 35s the patch passed
        +1 compile 0m 31s the patch passed with JDK v1.8.0_121
        +1 javac 0m 31s the patch passed
        +1 compile 0m 34s the patch passed with JDK v1.7.0_121
        +1 javac 0m 34s the patch passed
        -0 checkstyle 0m 33s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596)
        +1 mvnsite 0m 40s the patch passed
        +1 mvneclipse 0m 17s the patch passed
        +1 shellcheck 0m 8s There were no new shellcheck issues.
        +1 shelldocs 0m 13s There were no new shelldocs issues.
        +1 whitespace 0m 1s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 22s the patch passed with JDK v1.8.0_121
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_121
        -1 unit 41m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        119m 6s



        Reason Tests
        JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863521/YARN-6406-branch-2.002.patch
        Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 266f92a57b5d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 1699b20
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        shellcheck v0.4.6
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15648/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15648/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15648/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15648/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 42s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 7m 25s branch-2 passed +1 compile 0m 35s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 38s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 36s branch-2 passed +1 mvnsite 0m 43s branch-2 passed +1 mvneclipse 0m 22s branch-2 passed +1 findbugs 1m 21s branch-2 passed +1 javadoc 0m 25s branch-2 passed with JDK v1.8.0_121 +1 javadoc 0m 29s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 35s the patch passed +1 compile 0m 31s the patch passed with JDK v1.8.0_121 +1 javac 0m 31s the patch passed +1 compile 0m 34s the patch passed with JDK v1.7.0_121 +1 javac 0m 34s the patch passed -0 checkstyle 0m 33s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596) +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 17s the patch passed +1 shellcheck 0m 8s There were no new shellcheck issues. +1 shelldocs 0m 13s There were no new shelldocs issues. +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_121 -1 unit 41m 46s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 119m 6s Reason Tests JDK v1.8.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863521/YARN-6406-branch-2.002.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 266f92a57b5d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1699b20 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 shellcheck v0.4.6 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15648/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15648/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15648/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15648/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 15m 52s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 8m 29s branch-2 passed
        +1 compile 0m 36s branch-2 passed with JDK v1.8.0_121
        +1 compile 0m 39s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 43s branch-2 passed
        +1 mvnsite 0m 56s branch-2 passed
        +1 mvneclipse 0m 26s branch-2 passed
        +1 findbugs 1m 35s branch-2 passed
        +1 javadoc 0m 30s branch-2 passed with JDK v1.8.0_121
        +1 javadoc 0m 33s branch-2 passed with JDK v1.7.0_121
        +1 mvninstall 0m 40s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.8.0_121
        +1 javac 0m 39s the patch passed
        +1 compile 0m 37s the patch passed with JDK v1.7.0_121
        +1 javac 0m 37s the patch passed
        -0 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596)
        +1 mvnsite 0m 44s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 shellcheck 0m 10s There were no new shellcheck issues.
        +1 shelldocs 0m 10s There were no new shelldocs issues.
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 42s the patch passed
        +1 javadoc 0m 29s the patch passed with JDK v1.8.0_121
        +1 javadoc 0m 29s the patch passed with JDK v1.7.0_121
        -1 unit 39m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        118m 48s



        Reason Tests
        JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863521/YARN-6406-branch-2.002.patch
        Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 1a8b9cdaa910 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 1699b20
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        shellcheck v0.4.6
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15647/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15647/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15647/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15647/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 15m 52s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 8m 29s branch-2 passed +1 compile 0m 36s branch-2 passed with JDK v1.8.0_121 +1 compile 0m 39s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 43s branch-2 passed +1 mvnsite 0m 56s branch-2 passed +1 mvneclipse 0m 26s branch-2 passed +1 findbugs 1m 35s branch-2 passed +1 javadoc 0m 30s branch-2 passed with JDK v1.8.0_121 +1 javadoc 0m 33s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 40s the patch passed +1 compile 0m 39s the patch passed with JDK v1.8.0_121 +1 javac 0m 39s the patch passed +1 compile 0m 37s the patch passed with JDK v1.7.0_121 +1 javac 0m 37s the patch passed -0 checkstyle 0m 34s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596) +1 mvnsite 0m 44s the patch passed +1 mvneclipse 0m 34s the patch passed +1 shellcheck 0m 10s There were no new shellcheck issues. +1 shelldocs 0m 10s There were no new shelldocs issues. +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 42s the patch passed +1 javadoc 0m 29s the patch passed with JDK v1.8.0_121 +1 javadoc 0m 29s the patch passed with JDK v1.7.0_121 -1 unit 39m 45s hadoop-yarn-server-resourcemanager in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 118m 48s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863521/YARN-6406-branch-2.002.patch Optional Tests asflicense shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1a8b9cdaa910 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 1699b20 Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 shellcheck v0.4.6 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15647/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15647/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15647/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15647/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        A patch to the testing environment has been detected.
        Re-executing against the patched versions to perform further tests.
        The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15648/console in case of problems.

        Show
        hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15648/console in case of problems.
        Hide
        hadoopqa Hadoop QA added a comment -

        A patch to the testing environment has been detected.
        Re-executing against the patched versions to perform further tests.
        The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15647/console in case of problems.

        Show
        hadoopqa Hadoop QA added a comment - A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15647/console in case of problems.
        Hide
        asuresh Arun Suresh added a comment -

        Reattaching patch with HADOOP-14311 fix to see if Jenkins will work

        Show
        asuresh Arun Suresh added a comment - Reattaching patch with HADOOP-14311 fix to see if Jenkins will work
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 6s Docker failed to build yetus/hadoop:b59b8b7.



        Subsystem Report/Notes
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863411/YARN-6406-branch-2.001.patch
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15638/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 6s Docker failed to build yetus/hadoop:b59b8b7. Subsystem Report/Notes JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863411/YARN-6406-branch-2.001.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/15638/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Attaching patch for branch-2.
        Wangda Tan, can you take a look.

        Show
        asuresh Arun Suresh added a comment - Attaching patch for branch-2. Wangda Tan , can you take a look.
        Hide
        asuresh Arun Suresh added a comment -

        Reopening issue to add patch for branch-2

        Show
        asuresh Arun Suresh added a comment - Reopening issue to add patch for branch-2
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11531/)
        YARN-6406. Remove SchedulerRequestKeys when no more pending (wangda: rev 87e2ef8c985bb72a916477e8783359f2859f7890)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11531 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11531/ ) YARN-6406 . Remove SchedulerRequestKeys when no more pending (wangda: rev 87e2ef8c985bb72a916477e8783359f2859f7890) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
        Hide
        leftnoteasy Wangda Tan added a comment -

        Arun Suresh, I committed but forgot push to trunk yesterday. Just pushed to trunk.

        Show
        leftnoteasy Wangda Tan added a comment - Arun Suresh , I committed but forgot push to trunk yesterday. Just pushed to trunk.
        Hide
        asuresh Arun Suresh added a comment -

        Wangda Tan, for branch-2, I think we need to get YARN-6040 and YARN-6216 in first.
        do let me know once you have committed this to trunk.. and I can put a patch for branch-2

        Show
        asuresh Arun Suresh added a comment - Wangda Tan , for branch-2, I think we need to get YARN-6040 and YARN-6216 in first. do let me know once you have committed this to trunk.. and I can put a patch for branch-2
        Hide
        leftnoteasy Wangda Tan added a comment -

        Committed to trunk, thanks Arun Suresh and thanks reviews from Jason Lowe!

        Arun Suresh the patch doesn't apply to branch-2, could you update a patch against branch-2?

        Show
        leftnoteasy Wangda Tan added a comment - Committed to trunk, thanks Arun Suresh and thanks reviews from Jason Lowe ! Arun Suresh the patch doesn't apply to branch-2, could you update a patch against branch-2?
        Hide
        leftnoteasy Wangda Tan added a comment -

        LGTM, +1. Thanks Arun Suresh, will commit tomorrow.

        Show
        leftnoteasy Wangda Tan added a comment - LGTM, +1. Thanks Arun Suresh , will commit tomorrow.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 15m 38s trunk passed
        +1 compile 0m 32s trunk passed
        +1 checkstyle 0m 33s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 findbugs 1m 1s trunk passed
        +1 javadoc 0m 23s trunk passed
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 30s the patch passed
        +1 javac 0m 30s the patch passed
        -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 577 unchanged - 7 fixed = 584 total (was 584)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 38m 32s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        63m 8s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861764/YARN-6406.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux dfd376479e47 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5faa949
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15492/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15492/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15492/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15492/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 15m 38s trunk passed +1 compile 0m 32s trunk passed +1 checkstyle 0m 33s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 1s trunk passed +1 javadoc 0m 23s trunk passed +1 mvninstall 0m 33s the patch passed +1 compile 0m 30s the patch passed +1 javac 0m 30s the patch passed -0 checkstyle 0m 29s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 577 unchanged - 7 fixed = 584 total (was 584) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 38m 32s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 63m 8s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861764/YARN-6406.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dfd376479e47 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5faa949 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15492/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15492/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15492/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15492/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        asuresh Arun Suresh added a comment -

        Updating patch to revert changes to AppInfo and update the TestRMWebServicesApps test based on Wangda Tan's suggestion.

        Show
        asuresh Arun Suresh added a comment - Updating patch to revert changes to AppInfo and update the TestRMWebServicesApps test based on Wangda Tan 's suggestion.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Filed YARN-6429 and will work on that once this patch is done.

        Show
        leftnoteasy Wangda Tan added a comment - Filed YARN-6429 and will work on that once this patch is done.
        Hide
        leftnoteasy Wangda Tan added a comment -

        I think the right approach is to fix the test case (which might be harder).. thoughts ?

        I would suggest to fix the test case instead.

        Please feel free to open another JIRA for that (I can help review), but for the timebeing, I think we can remove the schedulerkey as is done in this patch ?

        Will do that.

        Show
        leftnoteasy Wangda Tan added a comment - I think the right approach is to fix the test case (which might be harder).. thoughts ? I would suggest to fix the test case instead. Please feel free to open another JIRA for that (I can help review), but for the timebeing, I think we can remove the schedulerkey as is done in this patch ? Will do that.
        Hide
        asuresh Arun Suresh added a comment -

        Thanks for the review Wangda Tan,

        Why changes of AppInfo required?

        Hmmm.. so the the TestRMWebServicesApps was complaining since it needed a resource request object. With this patch, if there is no outstanding resource request, The AppInfo will not contain any resource request objects. So I decided to send a dummy resourceRequest Object when none exist. I think the right approach is to fix the test case (which might be harder).. thoughts ?

        In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding ...

        Don't think it is a problem too much (based on existing code paths) .. but yes, maybe we should clean it up, since it could lead some circular references to the same placementset object.
        Please feel free to open another JIRA for that (I can help review), but for the timebeing, I think we can remove the schedulerkey as is done in this patch ?

        Show
        asuresh Arun Suresh added a comment - Thanks for the review Wangda Tan , Why changes of AppInfo required? Hmmm.. so the the TestRMWebServicesApps was complaining since it needed a resource request object. With this patch, if there is no outstanding resource request, The AppInfo will not contain any resource request objects. So I decided to send a dummy resourceRequest Object when none exist. I think the right approach is to fix the test case (which might be harder).. thoughts ? In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding ... Don't think it is a problem too much (based on existing code paths) .. but yes, maybe we should clean it up, since it could lead some circular references to the same placementset object. Please feel free to open another JIRA for that (I can help review), but for the timebeing, I think we can remove the schedulerkey as is done in this patch ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
        +1 mvninstall 13m 49s trunk passed
        +1 compile 0m 31s trunk passed
        +1 checkstyle 0m 32s trunk passed
        +1 mvnsite 0m 34s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 59s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        -0 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 622 unchanged - 6 fixed = 625 total (was 628)
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 7s the patch passed
        +1 javadoc 0m 22s the patch passed
        -1 unit 39m 16s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        61m 58s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6406
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861534/YARN-6406.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 10356231056e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 73835c7
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15461/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/15461/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15461/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15461/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. +1 mvninstall 13m 49s trunk passed +1 compile 0m 31s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 34s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 59s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 30s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 622 unchanged - 6 fixed = 625 total (was 628) +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 7s the patch passed +1 javadoc 0m 22s the patch passed -1 unit 39m 16s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 61m 58s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6406 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861534/YARN-6406.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 10356231056e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 73835c7 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15461/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15461/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15461/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15461/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Arun Suresh for working on the fix, my comments:

        1) Why changes of AppInfo required?
        2) Not caused by your patch (Actually caused by my patch). In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding, which could potentially cause trouble since it tries to modify parent from child. Is it possible to move this logic to AppSchedulingInfo#allocate. If it is a non trivial change, I can take it up in a separate JIRA.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Arun Suresh for working on the fix, my comments: 1) Why changes of AppInfo required? 2) Not caused by your patch (Actually caused by my patch). In LocalitySchedulingPlacementSet: it calls appSchedulingInfo directly in decrementOutstanding, which could potentially cause trouble since it tries to modify parent from child. Is it possible to move this logic to AppSchedulingInfo#allocate. If it is a non trivial change, I can take it up in a separate JIRA.
        Hide
        asuresh Arun Suresh added a comment -

        Attaching initial patch.

        • Moved the schedulerKeys dataStructure to a ConcurrentSkipListSet
        • Removed ANY ResourceRequests once num container == 0
        • Removed the SchedulerPlacementSet entry completely if no more resourceRequests exists
        • Added some tests.

        It looks like SchedulerKeys are actually removed once Containers are allocated. This patch additionally removes ResourceRequest objects which are SchedulerRequestKey key-ed entries as well.

        Show
        asuresh Arun Suresh added a comment - Attaching initial patch. Moved the schedulerKeys dataStructure to a ConcurrentSkipListSet Removed ANY ResourceRequests once num container == 0 Removed the SchedulerPlacementSet entry completely if no more resourceRequests exists Added some tests. It looks like SchedulerKeys are actually removed once Containers are allocated. This patch additionally removes ResourceRequest objects which are SchedulerRequestKey key-ed entries as well.
        Hide
        jlowe Jason Lowe added a comment -

        Yep, the refcount was only added because of the possibility of the two types of requests. When there are multiple refs to the key, we can't assume removing the last of one type removes all references to the key. If there is only one type that can reference the scheduler key then we don't need to refcount it separately.

        Show
        jlowe Jason Lowe added a comment - Yep, the refcount was only added because of the possibility of the two types of requests. When there are multiple refs to the key, we can't assume removing the last of one type removes all references to the key. If there is only one type that can reference the scheduler key then we don't need to refcount it separately.
        Hide
        asuresh Arun Suresh added a comment -

        Regarding to this, I don't think we need the #ref any more, correct?

        Yeah... think we can move this back to a set.

        Will post a patch with this shortly

        Show
        asuresh Arun Suresh added a comment - Regarding to this, I don't think we need the #ref any more, correct? Yeah... think we can move this back to a set. Will post a patch with this shortly
        Hide
        leftnoteasy Wangda Tan added a comment -

        Nice catch Arun Suresh.

        Arun Suresh/Jason Lowe, I totally agree to remove it once #pending-req = 0.

        Jason Lowe, IIRC, the reference number for scheduler key is added because we have two different request before: increase request and resource request. There's a change Arun Suresh did recently is, remove the increase request, now all increase request becomes a normal resource request. Regarding to this, I don't think we need the #ref any more, correct? To me a set of SchedulerKey will be good enough.

        Please share your thoughts.

        Show
        leftnoteasy Wangda Tan added a comment - Nice catch Arun Suresh . Arun Suresh / Jason Lowe , I totally agree to remove it once #pending-req = 0. Jason Lowe , IIRC, the reference number for scheduler key is added because we have two different request before: increase request and resource request. There's a change Arun Suresh did recently is, remove the increase request, now all increase request becomes a normal resource request. Regarding to this, I don't think we need the #ref any more, correct? To me a set of SchedulerKey will be good enough. Please share your thoughts.
        Hide
        jlowe Jason Lowe added a comment -

        I haven't dug into YARN-6040, but in general I'm a big +1 for having the RM aggressively remove bookkeeping entries that aren't necessary to improve lookup/iteration performance in addition to reducing the heap pressure. That was the whole idea behind YARN-5540. I don't see why we would need to keep scheduler keys or requests around once there are no more containers to allocate for them.

        Show
        jlowe Jason Lowe added a comment - I haven't dug into YARN-6040 , but in general I'm a big +1 for having the RM aggressively remove bookkeeping entries that aren't necessary to improve lookup/iteration performance in addition to reducing the heap pressure. That was the whole idea behind YARN-5540 . I don't see why we would need to keep scheduler keys or requests around once there are no more containers to allocate for them.
        Hide
        asuresh Arun Suresh added a comment -

        Wangda Tan / Jason Lowe Thoughts ?

        Show
        asuresh Arun Suresh added a comment - Wangda Tan / Jason Lowe Thoughts ?

          People

          • Assignee:
            asuresh Arun Suresh
            Reporter:
            asuresh Arun Suresh
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development