Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently AppSchedulingInfo simply stores resource request and scheduler make decision according to stored resource request. For example, CS/FS use slightly different approach to get pending resource request and make delay scheduling decision.

      There're several benefits of moving pending resource request data structure to SchedulingPlacementSet

      1) Delay scheduling logic should be agnostic to scheduler, for example CS supports count-based delay and FS supports both of count-based and time-based delay. Ideally scheduler should be able to choose which delay scheduling policy to use.
      2) In addition to 1., YARN-4902 has proposal to support pluggable delay scheduling behavior in addition to locality-based (host->rack->offswitch). Which requires more flexibility.
      3) To make YARN-4902 becomes real, instead of directly adding the new resource request API to client, we can make scheduler to use it internally to make sure it is well defined. And AppSchedulingInfo/SchedulingPlacementSet will be the perfect place to isolate which ResourceRequest implementation to use.
      4) Different scheduling requirement needs different behavior of checking ResourceRequest table.

      This JIRA is the 1st patch of several refactorings. Which moves all ResourceRequest data structure and logics to SchedulingPlacementSet. We need follow changes to make it better structured

      • Make delay scheduling to be a plugin of SchedulingPlacementSet
      • After YARN-4902 get committed, change SchedulingPlacementSet to use YARN-4902 internally.
      1. YARN-5906.1.patch
        39 kB
        Wangda Tan
      2. YARN-5906.2.patch
        39 kB
        Wangda Tan
      3. YARN-5906.3.patch
        41 kB
        Wangda Tan
      4. YARN-5906.4.patch
        41 kB
        Wangda Tan
      5. YARN-5906.5.patch
        41 kB
        Wangda Tan
      6. YARN-5906.branch-2.005.patch
        40 kB
        Wangda Tan

        Activity

        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks for backporting this! Sunil G!

        Show
        leftnoteasy Wangda Tan added a comment - Thanks for backporting this! Sunil G !
        Hide
        sunilg Sunil G added a comment -

        Committed to branch-2 as well. Thanks Wangda Tan

        Show
        sunilg Sunil G added a comment - Committed to branch-2 as well. Thanks Wangda Tan
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 25s branch-2 passed
        +1 compile 0m 32s branch-2 passed with JDK v1.8.0_111
        +1 compile 0m 35s branch-2 passed with JDK v1.7.0_121
        +1 checkstyle 0m 24s branch-2 passed
        +1 mvnsite 0m 40s branch-2 passed
        +1 mvneclipse 0m 18s branch-2 passed
        +1 findbugs 1m 21s branch-2 passed
        +1 javadoc 0m 23s branch-2 passed with JDK v1.8.0_111
        +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_121
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 29s the patch passed with JDK v1.8.0_111
        +1 javac 0m 29s the patch passed
        +1 compile 0m 33s the patch passed with JDK v1.7.0_121
        +1 javac 0m 33s the patch passed
        -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 46 unchanged - 2 fixed = 54 total (was 48)
        +1 mvnsite 0m 38s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 37s the patch passed
        +1 javadoc 0m 24s the patch passed with JDK v1.8.0_111
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_121
        +1 unit 41m 19s hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.7.0_121.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        101m 44s



        Reason Tests
        JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
          hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:b59b8b7
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845090/YARN-5906.branch-2.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 41ecd37a5f3c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / fd3b1ca
        Default Java 1.7.0_121
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14585/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14585/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14585/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 25s branch-2 passed +1 compile 0m 32s branch-2 passed with JDK v1.8.0_111 +1 compile 0m 35s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 24s branch-2 passed +1 mvnsite 0m 40s branch-2 passed +1 mvneclipse 0m 18s branch-2 passed +1 findbugs 1m 21s branch-2 passed +1 javadoc 0m 23s branch-2 passed with JDK v1.8.0_111 +1 javadoc 0m 26s branch-2 passed with JDK v1.7.0_121 +1 mvninstall 0m 34s the patch passed +1 compile 0m 29s the patch passed with JDK v1.8.0_111 +1 javac 0m 29s the patch passed +1 compile 0m 33s the patch passed with JDK v1.7.0_121 +1 javac 0m 33s the patch passed -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 46 unchanged - 2 fixed = 54 total (was 48) +1 mvnsite 0m 38s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 37s the patch passed +1 javadoc 0m 24s the patch passed with JDK v1.8.0_111 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_121 +1 unit 41m 19s hadoop-yarn-server-resourcemanager in the patch passed with JDK v1.7.0_121. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 101m 44s Reason Tests JDK v1.8.0_111 Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845090/YARN-5906.branch-2.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 41ecd37a5f3c 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / fd3b1ca Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_111 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14585/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14585/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14585/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        sunilg Sunil G added a comment -

        Kicking jenkins again for branch-2 patch.

        Show
        sunilg Sunil G added a comment - Kicking jenkins again for branch-2 patch.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached correct patch ..

        Show
        leftnoteasy Wangda Tan added a comment - Attached correct patch ..
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks Sunil G, since this is a refactoring patch, I prefer to commit to branch-2 as well to avoid divergence. Will attach branch-2 patch to run Jenkins soon.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks Sunil G , since this is a refactoring patch, I prefer to commit to branch-2 as well to avoid divergence. Will attach branch-2 patch to run Jenkins soon.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11050 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11050/)
        YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. (sunilg: rev 9ca54f4810de182195263bd594afb56dab564105)

        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimitsByPartition.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11050 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11050/ ) YARN-5906 . Update AppSchedulingInfo to use SchedulingPlacementSet. (sunilg: rev 9ca54f4810de182195263bd594afb56dab564105) (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SchedulingPlacementSet.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimitsByPartition.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java
        Hide
        sunilg Sunil G added a comment -

        Committed to trunk. Thanks Wangda Tan for contribution. I think this change is only for trunk, correct ?

        Show
        sunilg Sunil G added a comment - Committed to trunk. Thanks Wangda Tan for contribution. I think this change is only for trunk, correct ?
        Hide
        sunilg Sunil G added a comment -

        Test case failures are unrelated. Will commit in a short while.

        Show
        sunilg Sunil G added a comment - Test case failures are unrelated. Will commit in a short while.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 13m 10s trunk passed
        +1 compile 0m 33s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 36s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 9s trunk passed
        +1 javadoc 0m 24s trunk passed
        +1 mvninstall 0m 34s the patch passed
        +1 compile 0m 31s the patch passed
        +1 javac 0m 31s the patch passed
        -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 46 unchanged - 2 fixed = 54 total (was 48)
        +1 mvnsite 0m 42s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 3s the patch passed
        +1 javadoc 0m 18s the patch passed
        -1 unit 39m 35s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        61m 36s



        Reason Tests
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
          hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844876/YARN-5906.5.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 626ce8eb2e90 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0840b43
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14483/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14483/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14483/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14483/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 10s trunk passed +1 compile 0m 33s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 36s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 9s trunk passed +1 javadoc 0m 24s trunk passed +1 mvninstall 0m 34s the patch passed +1 compile 0m 31s the patch passed +1 javac 0m 31s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 8 new + 46 unchanged - 2 fixed = 54 total (was 48) +1 mvnsite 0m 42s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 3s the patch passed +1 javadoc 0m 18s the patch passed -1 unit 39m 35s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 61m 36s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844876/YARN-5906.5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 626ce8eb2e90 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0840b43 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14483/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14483/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14483/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14483/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.5 patch, rebased to latest trunk.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.5 patch, rebased to latest trunk.
        Hide
        sunilg Sunil G added a comment -

        Generally patch looks fine for me. I will commit tomorrow if there are no objections.

        Show
        sunilg Sunil G added a comment - Generally patch looks fine for me. I will commit tomorrow if there are no objections.
        Hide
        sunilg Sunil G added a comment -

        i might take a day more to take more closer look . Thank you.

        Show
        sunilg Sunil G added a comment - i might take a day more to take more closer look . Thank you.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 12m 57s trunk passed
        +1 compile 0m 35s trunk passed
        +1 checkstyle 0m 21s trunk passed
        +1 mvnsite 0m 37s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 5s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 32s the patch passed
        +1 compile 0m 32s the patch passed
        +1 javac 0m 32s the patch passed
        -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 46 unchanged - 2 fixed = 56 total (was 48)
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 10s the patch passed
        +1 javadoc 0m 20s the patch passed
        +1 unit 41m 0s hadoop-yarn-server-resourcemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        62m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844138/YARN-5906.4.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 46baaa01f892 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 736f54b
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14431/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14431/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14431/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 57s trunk passed +1 compile 0m 35s trunk passed +1 checkstyle 0m 21s trunk passed +1 mvnsite 0m 37s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 5s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 32s the patch passed +1 compile 0m 32s the patch passed +1 javac 0m 32s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 46 unchanged - 2 fixed = 56 total (was 48) +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 10s the patch passed +1 javadoc 0m 20s the patch passed +1 unit 41m 0s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 62m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12844138/YARN-5906.4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 46baaa01f892 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 736f54b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14431/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14431/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14431/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Sunil G,

        Thanks for comments,

        I was more or less thinking to move few common methods to an abstract class.

        I would prefer to do this once we have more SchedulingPlacementSet supported.

        My doubt is, when we need to provide an interator of a set of nodes which are arranged in some order (as per configuration policy), this code may become more trickier.

        What I want is implementing different sort mechanism inside different SchedulingPlacementSet, so I'm not sure if we need add a plugable iterator interface inside the schedulingPlacementSet, I would also prefer to do this when we adding more schedulingPlacementSet.

        Any other comments?

        Show
        leftnoteasy Wangda Tan added a comment - Sunil G , Thanks for comments, I was more or less thinking to move few common methods to an abstract class. I would prefer to do this once we have more SchedulingPlacementSet supported. My doubt is, when we need to provide an interator of a set of nodes which are arranged in some order (as per configuration policy), this code may become more trickier. What I want is implementing different sort mechanism inside different SchedulingPlacementSet, so I'm not sure if we need add a plugable iterator interface inside the schedulingPlacementSet, I would also prefer to do this when we adding more schedulingPlacementSet. Any other comments?
        Hide
        sunilg Sunil G added a comment -

        Were you suggesting to move some implementations to an abstract base class to share between other implementations, or were you suggesting to move some private methods to the SchedulingPlacementSet interface?

        I was more or less thinking to move few common methods to an abstract class.

        Not sure if I understand the question, different SchedulingPlacementSet could have different ordering of preferred node. Could you elaborate?

        LocalitySchedulingPlacementSet#getPreferredNodeIterator currently trying to get singleNode from PlacementSetUtils.getSingleNode(clusterPlacementSet) call. Going forward, if we need to have different policies to select nodes for different allocations, i think we will be invoking getSingleNode or getMultiNode etc. My doubt is, when we need to provide an interator of a set of nodes which are arranged in some order (as per configuration policy), this code may become more trickier. Could we do an interface model here, so that different node ordering policies could be injected?

        Show
        sunilg Sunil G added a comment - Were you suggesting to move some implementations to an abstract base class to share between other implementations, or were you suggesting to move some private methods to the SchedulingPlacementSet interface? I was more or less thinking to move few common methods to an abstract class. Not sure if I understand the question, different SchedulingPlacementSet could have different ordering of preferred node. Could you elaborate? LocalitySchedulingPlacementSet#getPreferredNodeIterator currently trying to get singleNode from PlacementSetUtils.getSingleNode(clusterPlacementSet) call. Going forward, if we need to have different policies to select nodes for different allocations, i think we will be invoking getSingleNode or getMultiNode etc. My doubt is, when we need to provide an interator of a set of nodes which are arranged in some order (as per configuration policy), this code may become more trickier. Could we do an interface model here, so that different node ordering policies could be injected?
        Hide
        leftnoteasy Wangda Tan added a comment -

        Thanks reviews from Sunil G.

        LocalitySchedulingPlacementSet implements few common placementset apis

        Were you suggesting to move some implementations to an abstract base class to share between other implementations, or were you suggesting to move some private methods to the SchedulingPlacementSet interface?

        3. AppSchedulingInfo#getResourceRequests return empty map ...

        The major reason is to avoid null pointer check, and the emptyMap is a static variable so it should not have any performance issue. Do you is it necessary to expose "null" map to external caller?

        LocalitySchedulingPlacementSet has writelock and readlock ...

        I can elaborate more: In my mind, SchedulingPlacementSet could be accessed externally, for example, if we have any resource update, or container from other application completed (for example inter-app antiaffinity). We can notify the SchedulingPlacementSet. I think using the same AppSchedulingInfo may not good enough. it should not risky because it doesn't invoke methods from any other classes.

        5. LocalitySchedulingPlacementSet supports singleNode from ..

        Not sure if I understand the question, different SchedulingPlacementSet could have different ordering of preferred node. Could you elaborate?

        Attaching ver.4 patch which addressed #1 with some minor updates.

        Show
        leftnoteasy Wangda Tan added a comment - Thanks reviews from Sunil G . LocalitySchedulingPlacementSet implements few common placementset apis Were you suggesting to move some implementations to an abstract base class to share between other implementations, or were you suggesting to move some private methods to the SchedulingPlacementSet interface? 3. AppSchedulingInfo#getResourceRequests return empty map ... The major reason is to avoid null pointer check, and the emptyMap is a static variable so it should not have any performance issue. Do you is it necessary to expose "null" map to external caller? LocalitySchedulingPlacementSet has writelock and readlock ... I can elaborate more: In my mind, SchedulingPlacementSet could be accessed externally, for example, if we have any resource update, or container from other application completed (for example inter-app antiaffinity). We can notify the SchedulingPlacementSet. I think using the same AppSchedulingInfo may not good enough. it should not risky because it doesn't invoke methods from any other classes. 5. LocalitySchedulingPlacementSet supports singleNode from .. Not sure if I understand the question, different SchedulingPlacementSet could have different ordering of preferred node. Could you elaborate? Attaching ver.4 patch which addressed #1 with some minor updates.
        Hide
        sunilg Sunil G added a comment -

        Thanks Wangda Tan for the patch. Generally approach seems fine for me.

        Few comments:

        1. getSchedulingPlacementSet Remove the ToDO comments as a clean impl is done with this patch.

        2. LocalitySchedulingPlacementSet implements few common placementset apis'. Do you feel few of them could be moved to abstract class and could keep the abstractness ready if more type of placements could be configured as policy later.

        3. AppSchedulingInfo#getResourceRequests return empty map when placement set comes null fr given SchedulerKey. We could return null to the caller instead of an empty map, correct? any specific reasons?

        4. LocalitySchedulingPlacementSet has writelock and readlock. We operate on AppSchedulingInfo from FiCaSchedulerApp, so after this new lock, it will be a 3rd layer of lock. will it be risky? could we operate under AppSchedulingInfo lock itself?

        5. LocalitySchedulingPlacementSet supports singleNode from getPreferredNodeIterator now. As we go on, if different policy based iterator is needed, how can we attach policy to LocalitySchedulingPlacementSet?

        Show
        sunilg Sunil G added a comment - Thanks Wangda Tan for the patch. Generally approach seems fine for me. Few comments: 1. getSchedulingPlacementSet Remove the ToDO comments as a clean impl is done with this patch. 2. LocalitySchedulingPlacementSet implements few common placementset apis'. Do you feel few of them could be moved to abstract class and could keep the abstractness ready if more type of placements could be configured as policy later. 3. AppSchedulingInfo#getResourceRequests return empty map when placement set comes null fr given SchedulerKey. We could return null to the caller instead of an empty map, correct? any specific reasons? 4. LocalitySchedulingPlacementSet has writelock and readlock. We operate on AppSchedulingInfo from FiCaSchedulerApp, so after this new lock, it will be a 3rd layer of lock. will it be risky? could we operate under AppSchedulingInfo lock itself? 5. LocalitySchedulingPlacementSet supports singleNode from getPreferredNodeIterator now. As we go on, if different policy based iterator is needed, how can we attach policy to LocalitySchedulingPlacementSet ?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 23s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 15s trunk passed
        +1 compile 0m 40s trunk passed
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 0m 45s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 6s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 39s the patch passed
        +1 compile 0m 37s the patch passed
        +1 javac 0m 37s the patch passed
        -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 46 unchanged - 2 fixed = 56 total (was 48)
        +1 mvnsite 0m 44s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 20s the patch passed
        +1 javadoc 0m 24s the patch passed
        -1 unit 40m 5s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        58m 18s



        Reason Tests
        Failed junit tests hadoop.yarn.webapp.TestRMWithCSRFFilter
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
          hadoop.yarn.server.resourcemanager.TestResourceTrackerService
          hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842397/YARN-5906.3.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b41a90afe4d1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c265515
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14232/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14232/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 15s trunk passed +1 compile 0m 40s trunk passed +1 checkstyle 0m 23s trunk passed +1 mvnsite 0m 45s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 6s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 39s the patch passed +1 compile 0m 37s the patch passed +1 javac 0m 37s the patch passed -0 checkstyle 0m 21s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 10 new + 46 unchanged - 2 fixed = 56 total (was 48) +1 mvnsite 0m 44s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 20s the patch passed +1 javadoc 0m 24s the patch passed -1 unit 40m 5s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 58m 18s Reason Tests Failed junit tests hadoop.yarn.webapp.TestRMWithCSRFFilter   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices   hadoop.yarn.server.resourcemanager.TestResourceTrackerService   hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842397/YARN-5906.3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b41a90afe4d1 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c265515 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14232/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14232/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14232/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14232/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Uploaded correct patch (ver.3)

        Show
        leftnoteasy Wangda Tan added a comment - Uploaded correct patch (ver.3)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 35s trunk passed
        +1 compile 0m 37s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 47s trunk passed
        +1 mvneclipse 0m 18s trunk passed
        +1 findbugs 1m 15s trunk passed
        +1 javadoc 0m 24s trunk passed
        -1 mvninstall 0m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 compile 0m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        -1 javac 0m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 46 unchanged - 2 fixed = 59 total (was 48)
        -1 mvnsite 0m 24s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 mvneclipse 0m 17s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 0m 21s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 javadoc 0m 23s the patch passed
        -1 unit 0m 23s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        16m 55s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842390/YARN-5906.2.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 593985236af2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c265515
        Default Java 1.8.0_111
        findbugs v3.0.0
        mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        compile https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14229/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14229/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 35s trunk passed +1 compile 0m 37s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 47s trunk passed +1 mvneclipse 0m 18s trunk passed +1 findbugs 1m 15s trunk passed +1 javadoc 0m 24s trunk passed -1 mvninstall 0m 24s hadoop-yarn-server-resourcemanager in the patch failed. -1 compile 0m 23s hadoop-yarn-server-resourcemanager in the patch failed. -1 javac 0m 23s hadoop-yarn-server-resourcemanager in the patch failed. -0 checkstyle 0m 22s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 46 unchanged - 2 fixed = 59 total (was 48) -1 mvnsite 0m 24s hadoop-yarn-server-resourcemanager in the patch failed. +1 mvneclipse 0m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 21s hadoop-yarn-server-resourcemanager in the patch failed. +1 javadoc 0m 23s the patch passed -1 unit 0m 23s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 16m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12842390/YARN-5906.2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 593985236af2 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c265515 Default Java 1.8.0_111 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/14229/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14229/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/14229/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.2 patch, rebased to latest trunk.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.2 patch, rebased to latest trunk.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 11s trunk passed
        +1 compile 0m 34s trunk passed
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 39s trunk passed
        +1 mvneclipse 0m 17s trunk passed
        +1 findbugs 1m 3s trunk passed
        +1 javadoc 0m 22s trunk passed
        +1 mvninstall 0m 33s the patch passed
        +1 compile 0m 34s the patch passed
        +1 javac 0m 34s the patch passed
        -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 46 unchanged - 2 fixed = 59 total (was 48)
        +1 mvnsite 0m 40s the patch passed
        +1 mvneclipse 0m 15s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 0m 21s the patch passed
        -1 unit 46m 10s hadoop-yarn-server-resourcemanager in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        62m 26s



        Reason Tests
        FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(List, boolean) makes inefficient use of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:[line 428]
        Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5906
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839447/YARN-5906.1.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a2d8a0cf12dc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / bd37355
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13961/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13961/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 11s trunk passed +1 compile 0m 34s trunk passed +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 39s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 3s trunk passed +1 javadoc 0m 22s trunk passed +1 mvninstall 0m 33s the patch passed +1 compile 0m 34s the patch passed +1 javac 0m 34s the patch passed -0 checkstyle 0m 20s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 46 unchanged - 2 fixed = 59 total (was 48) +1 mvnsite 0m 40s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 21s the patch passed -1 unit 46m 10s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 62m 26s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager   org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(List, boolean) makes inefficient use of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java:of keySet iterator instead of entrySet iterator At AppSchedulingInfo.java: [line 428] Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5906 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839447/YARN-5906.1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a2d8a0cf12dc 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bd37355 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/13961/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13961/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13961/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        leftnoteasy Wangda Tan added a comment -

        Attached ver.1 patch for review.

        + Jian He, Karthik Kambatla, Arun Suresh, Subru Krishnan please feel free to share your thoughts for the patch and overall plan.

        Show
        leftnoteasy Wangda Tan added a comment - Attached ver.1 patch for review. + Jian He , Karthik Kambatla , Arun Suresh , Subru Krishnan please feel free to share your thoughts for the patch and overall plan.

          People

          • Assignee:
            leftnoteasy Wangda Tan
            Reporter:
            leftnoteasy Wangda Tan
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development