Details

    • Hadoop Flags:
      Reviewed

      Description

      One of the main tenets the YARN Federation is to transparently scale applications across multiple clusters. This is achieved by running UAMs on behalf of the application on other clusters. This JIRA tracks the addition of a UnmanagedAM pool manager for federating application across clusters which will be used the FederationInterceptor (YARN-3666) which is part of the AMRMProxy pipeline introduced in YARN-2884.

      1. YARN-5531-YARN-2915.v1.patch
        55 kB
        Botong Huang
      2. YARN-5531-YARN-2915.v10.patch
        82 kB
        Botong Huang
      3. YARN-5531-YARN-2915.v11.patch
        81 kB
        Botong Huang
      4. YARN-5531-YARN-2915.v12.patch
        81 kB
        Botong Huang
      5. YARN-5531-YARN-2915.v13.patch
        86 kB
        Botong Huang
      6. YARN-5531-YARN-2915.v14.patch
        86 kB
        Botong Huang
      7. YARN-5531-YARN-2915.v15.patch
        86 kB
        Botong Huang
      8. YARN-5531-YARN-2915.v2.patch
        55 kB
        Botong Huang
      9. YARN-5531-YARN-2915.v3.patch
        57 kB
        Botong Huang
      10. YARN-5531-YARN-2915.v4.patch
        70 kB
        Botong Huang
      11. YARN-5531-YARN-2915.v5.patch
        90 kB
        Botong Huang
      12. YARN-5531-YARN-2915.v6.patch
        90 kB
        Botong Huang
      13. YARN-5531-YARN-2915.v7.patch
        83 kB
        Botong Huang
      14. YARN-5531-YARN-2915.v8.patch
        82 kB
        Botong Huang
      15. YARN-5531-YARN-2915.v9.patch
        82 kB
        Botong Huang

        Issue Links

          Activity

          Hide
          kasha Karthik Kambatla added a comment -

          I had filed YARN-2963 at some point to expose an Unmanaged AM pool. Any chance we could expose this construct outside of federation? I am not sure how much of my code on YARN-2963 is useful anymore, but feel free to use it as necessary. Happy to review this.

          Show
          kasha Karthik Kambatla added a comment - I had filed YARN-2963 at some point to expose an Unmanaged AM pool. Any chance we could expose this construct outside of federation? I am not sure how much of my code on YARN-2963 is useful anymore, but feel free to use it as necessary. Happy to review this.
          Hide
          botong Botong Huang added a comment -

          This is the initial version of UnmanagedAM, written for Federation. However we tried to stay generic so that it can be used for other scenarios as well.

          Show
          botong Botong Huang added a comment - This is the initial version of UnmanagedAM, written for Federation. However we tried to stay generic so that it can be used for other scenarios as well.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 17m 6s YARN-2915 passed
          -1 compile 5m 24s hadoop-yarn in YARN-2915 failed.
          +1 checkstyle 0m 59s YARN-2915 passed
          +1 mvnsite 1m 16s YARN-2915 passed
          +1 mvneclipse 0m 47s YARN-2915 passed
          +1 findbugs 2m 15s YARN-2915 passed
          +1 javadoc 1m 7s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 52s the patch passed
          -1 compile 5m 6s hadoop-yarn in the patch failed.
          -1 javac 5m 6s hadoop-yarn in the patch failed.
          -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0)
          +1 mvnsite 1m 16s the patch passed
          +1 mvneclipse 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 18s the patch passed
          -1 javadoc 0m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 2 new + 4575 unchanged - 0 fixed = 4577 total (was 4575)
          -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 20 new + 162 unchanged - 0 fixed = 182 total (was 162)
          +1 unit 2m 36s hadoop-yarn-common in the patch passed.
          +1 unit 1m 30s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          56m 17s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861611/YARN-5531-YARN-2915.v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4adec98079ca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 39d352c
          Default Java 1.8.0_121
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15472/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15472/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 17m 6s YARN-2915 passed -1 compile 5m 24s hadoop-yarn in YARN-2915 failed. +1 checkstyle 0m 59s YARN-2915 passed +1 mvnsite 1m 16s YARN-2915 passed +1 mvneclipse 0m 47s YARN-2915 passed +1 findbugs 2m 15s YARN-2915 passed +1 javadoc 1m 7s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 52s the patch passed -1 compile 5m 6s hadoop-yarn in the patch failed. -1 javac 5m 6s hadoop-yarn in the patch failed. -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 18 new + 0 unchanged - 0 fixed = 18 total (was 0) +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 18s the patch passed -1 javadoc 0m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 2 new + 4575 unchanged - 0 fixed = 4577 total (was 4575) -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 20 new + 162 unchanged - 0 fixed = 182 total (was 162) +1 unit 2m 36s hadoop-yarn-common in the patch passed. +1 unit 1m 30s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 56m 17s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861611/YARN-5531-YARN-2915.v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4adec98079ca 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 39d352c Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15472/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15472/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15472/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 12m 40s YARN-2915 passed
          -1 compile 4m 51s hadoop-yarn in YARN-2915 failed.
          +1 checkstyle 0m 57s YARN-2915 passed
          +1 mvnsite 1m 15s YARN-2915 passed
          +1 mvneclipse 0m 48s YARN-2915 passed
          +1 findbugs 2m 4s YARN-2915 passed
          +1 javadoc 1m 8s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 47s the patch passed
          -1 compile 4m 32s hadoop-yarn in the patch failed.
          -1 javac 4m 32s hadoop-yarn in the patch failed.
          -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 mvnsite 1m 14s the patch passed
          +1 mvneclipse 0m 47s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 17s the patch passed
          -1 javadoc 0m 38s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 2 new + 4575 unchanged - 0 fixed = 4577 total (was 4575)
          -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 23 new + 162 unchanged - 0 fixed = 185 total (was 162)
          +1 unit 2m 30s hadoop-yarn-common in the patch passed.
          +1 unit 1m 26s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 41s The patch does not generate ASF License warnings.
          50m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861644/YARN-5531-YARN-2915.v2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 588cad9f341f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 39d352c
          Default Java 1.8.0_121
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
          findbugs v3.0.0
          compile https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15482/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15482/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 12m 40s YARN-2915 passed -1 compile 4m 51s hadoop-yarn in YARN-2915 failed. +1 checkstyle 0m 57s YARN-2915 passed +1 mvnsite 1m 15s YARN-2915 passed +1 mvneclipse 0m 48s YARN-2915 passed +1 findbugs 2m 4s YARN-2915 passed +1 javadoc 1m 8s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 47s the patch passed -1 compile 4m 32s hadoop-yarn in the patch failed. -1 javac 4m 32s hadoop-yarn in the patch failed. -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 mvnsite 1m 14s the patch passed +1 mvneclipse 0m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 17s the patch passed -1 javadoc 0m 38s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 2 new + 4575 unchanged - 0 fixed = 4577 total (was 4575) -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 23 new + 162 unchanged - 0 fixed = 185 total (was 162) +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 1m 26s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 41s The patch does not generate ASF License warnings. 50m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861644/YARN-5531-YARN-2915.v2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 588cad9f341f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 39d352c Default Java 1.8.0_121 compile https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt findbugs v3.0.0 compile https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15482/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15482/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15482/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 54s Maven dependency ordering for branch
          +1 mvninstall 19m 29s YARN-2915 passed
          +1 compile 11m 12s YARN-2915 passed
          +1 checkstyle 0m 52s YARN-2915 passed
          +1 mvnsite 1m 7s YARN-2915 passed
          +1 mvneclipse 0m 40s YARN-2915 passed
          +1 findbugs 2m 0s YARN-2915 passed
          +1 javadoc 0m 59s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 0m 54s the patch passed
          +1 compile 9m 23s the patch passed
          +1 javac 9m 23s the patch passed
          +1 checkstyle 1m 3s the patch passed
          +1 mvnsite 1m 31s the patch passed
          +1 mvneclipse 1m 6s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 29s the patch passed
          -1 javadoc 0m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 9 new + 162 unchanged - 0 fixed = 171 total (was 162)
          +1 unit 3m 34s hadoop-yarn-common in the patch passed.
          +1 unit 1m 45s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 32s The patch does not generate ASF License warnings.
          71m 57s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861771/YARN-5531-YARN-2915.v3.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a939bb676bc4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 6e34518
          Default Java 1.8.0_121
          findbugs v3.0.0
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15509/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15509/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15509/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 54s Maven dependency ordering for branch +1 mvninstall 19m 29s YARN-2915 passed +1 compile 11m 12s YARN-2915 passed +1 checkstyle 0m 52s YARN-2915 passed +1 mvnsite 1m 7s YARN-2915 passed +1 mvneclipse 0m 40s YARN-2915 passed +1 findbugs 2m 0s YARN-2915 passed +1 javadoc 0m 59s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 0m 54s the patch passed +1 compile 9m 23s the patch passed +1 javac 9m 23s the patch passed +1 checkstyle 1m 3s the patch passed +1 mvnsite 1m 31s the patch passed +1 mvneclipse 1m 6s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 29s the patch passed -1 javadoc 0m 40s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 9 new + 162 unchanged - 0 fixed = 171 total (was 162) +1 unit 3m 34s hadoop-yarn-common in the patch passed. +1 unit 1m 45s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 32s The patch does not generate ASF License warnings. 71m 57s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861771/YARN-5531-YARN-2915.v3.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a939bb676bc4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 6e34518 Default Java 1.8.0_121 findbugs v3.0.0 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15509/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15509/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15509/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Barely skimmed through the patch. Some high-level comments:

          1. Looks like we are tracking, with booleans, if the app is registered/finished. Would AbstractService be appropriate for this? Callers could init, start and stop? The downside would be init doesn't return RegisterApplicationMasterResponse.
          2. Logging could use more details. For instance, application_id.
          Show
          kasha Karthik Kambatla added a comment - Barely skimmed through the patch. Some high-level comments: Looks like we are tracking, with booleans, if the app is registered/finished. Would AbstractService be appropriate for this? Callers could init, start and stop? The downside would be init doesn't return RegisterApplicationMasterResponse . Logging could use more details. For instance, application_id.
          Hide
          botong Botong Huang added a comment - - edited

          Thanks Karthik Kambatla for the review! For 2, sure I will add more logs. For 1, to me this class is more like a library rather than a service, which hides the fact that this is an UAM in the remote Yarn cluster, and present an aync version of ApplicationMasterProtocol interface to user (the actual AM). We can modify it into a service as well if needed.

          Subru Krishnan: along this line of thought, maybe we should call this class UnmanagedApplicationMasterProxy? Thanks!

          Show
          botong Botong Huang added a comment - - edited Thanks Karthik Kambatla for the review! For 2, sure I will add more logs. For 1, to me this class is more like a library rather than a service, which hides the fact that this is an UAM in the remote Yarn cluster, and present an aync version of ApplicationMasterProtocol interface to user (the actual AM). We can modify it into a service as well if needed. Subru Krishnan : along this line of thought, maybe we should call this class UnmanagedApplicationMasterProxy? Thanks!
          Hide
          botong Botong Huang added a comment -

          UAM pool manager added before UAM

          Show
          botong Botong Huang added a comment - UAM pool manager added before UAM
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for the patch. I went through it, please find my feedback below.

          UnmanagedAMPoolManager:

          • We should Initialize threadpool in default constructor or in serviceStart (if null).
          • The order of operations in finishApplicationMaster should be reversed as now client retries are not possible?

          UnmanagedApplicationMaster:

          • Nit: rename UnmanagedApplicationMaster --> UnmanagedApplicationManager.
          • IIUC, requestQueuePendingCount is redundant, we should use requestQueue.size.
          • registerUAMCalled is also redundant as we can get the same information from lastResponseId or registerRequest.
          • A code comment explaining when userUgi will be null/reused will be useful.
          • Shouldn't forceKillApplication check for Am registered state?
          • The createRMProxy is very similar to the one in FederationProxyProviderUtil, refactor it out to YarnServerSecurityUtils and reuse.
          • Same comment for updateAMRMToken

          CallbackHandlerThread

          • There are quite a few throw RuntimeException which will exit the thread. Do we have any better options or is this fine?

          UnmanagedAMLauncher

          • Can we reuse code in submitAppAndGetAppId, monitorApplicationSubmission, monitorCurrentAppAttempt & getApplicationReport with the existing UAM launcher here?
          • Why do we need the attemptId in both the constructor initializeUnmanagedAM as they seem to invoked subsequently? Addressing this should also cleanup the TODO in initializeUnmanagedAM.
          • The initialization of recordFactory is redundant in submitAppAndGetAppId.

          Tests

          • MockResourceManagerFacade already exists here, so we should reuse instead of adding it again.
          • There are no tests for forceKillApplication and re-registering.
          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for the patch. I went through it, please find my feedback below. UnmanagedAMPoolManager : We should Initialize threadpool in default constructor or in serviceStart (if null). The order of operations in finishApplicationMaster should be reversed as now client retries are not possible? UnmanagedApplicationMaster : Nit: rename UnmanagedApplicationMaster --> UnmanagedApplicationManager . IIUC, requestQueuePendingCount is redundant, we should use requestQueue.size . registerUAMCalled is also redundant as we can get the same information from lastResponseId or registerRequest . A code comment explaining when userUgi will be null/reused will be useful. Shouldn't forceKillApplication check for Am registered state? The createRMProxy is very similar to the one in FederationProxyProviderUtil , refactor it out to YarnServerSecurityUtils and reuse. Same comment for updateAMRMToken CallbackHandlerThread There are quite a few throw RuntimeException which will exit the thread. Do we have any better options or is this fine? UnmanagedAMLauncher Can we reuse code in submitAppAndGetAppId , monitorApplicationSubmission , monitorCurrentAppAttempt & getApplicationReport with the existing UAM launcher here ? Why do we need the attemptId in both the constructor initializeUnmanagedAM as they seem to invoked subsequently? Addressing this should also cleanup the TODO in initializeUnmanagedAM . The initialization of recordFactory is redundant in submitAppAndGetAppId . Tests MockResourceManagerFacade already exists here , so we should reuse instead of adding it again. There are no tests for forceKillApplication and re-registering.
          Hide
          botong Botong Huang added a comment - - edited

          Thanks Subru Krishnan for the comments! All except the following is fixed in v5 patch:

          • We should Initialize threadpool in default constructor or in serviceStart (if null)
            • threadpool is not a Yarn service, and no serviceStart call
          • Shouldn't forceKillApplication check for Am registered state?
            • forceKillApplication is in ApplicationClientProtocol, it simply kills the app if there.
          • There are quite a few throw RuntimeException which will exit the thread. Do we have any better options or is this fine?
            • There's a catch Throwable at last
          • Can we reuse code in submitAppAndGetAppId, monitorApplicationSubmission, monitorCurrentAppAttempt & getApplicationReport with the existing UAM launcher here?
            • It is a bit different because it goes throw YarnClient. There's quite some modifications needed to reuse that code.
          Show
          botong Botong Huang added a comment - - edited Thanks Subru Krishnan for the comments! All except the following is fixed in v5 patch: We should Initialize threadpool in default constructor or in serviceStart (if null) threadpool is not a Yarn service, and no serviceStart call Shouldn't forceKillApplication check for Am registered state? forceKillApplication is in ApplicationClientProtocol , it simply kills the app if there. There are quite a few throw RuntimeException which will exit the thread. Do we have any better options or is this fine? There's a catch Throwable at last Can we reuse code in submitAppAndGetAppId, monitorApplicationSubmission, monitorCurrentAppAttempt & getApplicationReport with the existing UAM launcher here? It is a bit different because it goes throw YarnClient. There's quite some modifications needed to reuse that code.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 1m 40s Maven dependency ordering for branch
          +1 mvninstall 16m 2s YARN-2915 passed
          +1 compile 16m 35s YARN-2915 passed
          +1 checkstyle 1m 55s YARN-2915 passed
          +1 mvnsite 3m 11s YARN-2915 passed
          +1 mvneclipse 2m 13s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 0m 40s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings.
          +1 javadoc 2m 39s YARN-2915 passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 2m 27s the patch passed
          +1 compile 14m 30s the patch passed
          +1 javac 14m 30s the patch passed
          -0 checkstyle 1m 58s root: The patch generated 12 new + 48 unchanged - 1 fixed = 60 total (was 49)
          +1 mvnsite 3m 29s the patch passed
          +1 mvneclipse 2m 30s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 5m 27s the patch passed
          +1 javadoc 2m 59s the patch passed
          +1 unit 0m 21s hadoop-project in the patch passed.
          +1 unit 2m 31s hadoop-yarn-common in the patch passed.
          +1 unit 1m 25s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 38m 46s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 unit 19m 29s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 43s The patch does not generate ASF License warnings.
          185m 0s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866524/YARN-5531-YARN-2915.v5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 8d677b44a1da 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 9ffb5d2
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15834/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15834/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 1m 40s Maven dependency ordering for branch +1 mvninstall 16m 2s YARN-2915 passed +1 compile 16m 35s YARN-2915 passed +1 checkstyle 1m 55s YARN-2915 passed +1 mvnsite 3m 11s YARN-2915 passed +1 mvneclipse 2m 13s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 0m 40s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings. +1 javadoc 2m 39s YARN-2915 passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 2m 27s the patch passed +1 compile 14m 30s the patch passed +1 javac 14m 30s the patch passed -0 checkstyle 1m 58s root: The patch generated 12 new + 48 unchanged - 1 fixed = 60 total (was 49) +1 mvnsite 3m 29s the patch passed +1 mvneclipse 2m 30s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 5m 27s the patch passed +1 javadoc 2m 59s the patch passed +1 unit 0m 21s hadoop-project in the patch passed. +1 unit 2m 31s hadoop-yarn-common in the patch passed. +1 unit 1m 25s hadoop-yarn-server-common in the patch passed. +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 38m 46s hadoop-yarn-server-resourcemanager in the patch failed. +1 unit 19m 29s hadoop-yarn-client in the patch passed. +1 asflicense 0m 43s The patch does not generate ASF License warnings. 185m 0s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866524/YARN-5531-YARN-2915.v5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 8d677b44a1da 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 9ffb5d2 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15834/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15834/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15834/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 36s Maven dependency ordering for branch
          +1 mvninstall 16m 54s YARN-2915 passed
          +1 compile 17m 14s YARN-2915 passed
          +1 checkstyle 1m 58s YARN-2915 passed
          +1 mvnsite 3m 16s YARN-2915 passed
          +1 mvneclipse 2m 14s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings.
          +1 javadoc 2m 42s YARN-2915 passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 2m 36s the patch passed
          +1 compile 15m 38s the patch passed
          +1 javac 15m 38s the patch passed
          +1 checkstyle 2m 13s root: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49)
          +1 mvnsite 3m 52s the patch passed
          +1 mvneclipse 2m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 6m 17s the patch passed
          +1 javadoc 3m 15s the patch passed
          +1 unit 0m 20s hadoop-project in the patch passed.
          +1 unit 2m 47s hadoop-yarn-common in the patch passed.
          +1 unit 1m 37s hadoop-yarn-server-common in the patch passed.
          -1 unit 30m 7s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 39m 17s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 unit 20m 7s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          207m 31s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
            hadoop.yarn.server.nodemanager.TestNodeManagerReboot
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
          Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866649/YARN-5531-YARN-2915.v6.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 5dea6100bfdc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 9ffb5d2
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15839/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15839/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 36s Maven dependency ordering for branch +1 mvninstall 16m 54s YARN-2915 passed +1 compile 17m 14s YARN-2915 passed +1 checkstyle 1m 58s YARN-2915 passed +1 mvnsite 3m 16s YARN-2915 passed +1 mvneclipse 2m 14s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. -1 findbugs 1m 8s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings. +1 javadoc 2m 42s YARN-2915 passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 2m 36s the patch passed +1 compile 15m 38s the patch passed +1 javac 15m 38s the patch passed +1 checkstyle 2m 13s root: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49) +1 mvnsite 3m 52s the patch passed +1 mvneclipse 2m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 6m 17s the patch passed +1 javadoc 3m 15s the patch passed +1 unit 0m 20s hadoop-project in the patch passed. +1 unit 2m 47s hadoop-yarn-common in the patch passed. +1 unit 1m 37s hadoop-yarn-server-common in the patch passed. -1 unit 30m 7s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 39m 17s hadoop-yarn-server-resourcemanager in the patch passed. +1 unit 20m 7s hadoop-yarn-client in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 207m 31s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer   hadoop.yarn.server.nodemanager.TestNodeStatusUpdater   hadoop.yarn.server.nodemanager.TestNodeManagerReboot   hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866649/YARN-5531-YARN-2915.v6.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 5dea6100bfdc 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 9ffb5d2 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/15839/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15839/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/15839/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for addressing my comments. I looked at the latest patch is pretty close, have a few comments:

          • The pom changes for hadoop-yarn-server-common test dependencies are already committed as part of YARN-5411, can you kindly rebase.
          • Don't remove FederationProxyProviderUtil, we need it to manage Federation dependencies which currently has leaked into YarnServerSecurityUtils. Maybe AMRMProxyServiceUtils is a better place to put the createRMProxy but either ways the general utility class should not have any dependency to Federation.
          • hreadpool is not a Yarn service, and no serviceStart call

            I don't get your comment as the threadpool is managed by UnmanagedAMPoolManager which is indeed a service:

             public class UnmanagedAMPoolManager extends AbstractService 
          • UnmanagedAMLauncher::submitAppAndGetAppId can be void.
          •  if (!appAttemptId.equals(this.attemptId)) { 

            - I feel this check is not correct as once we have HA (YARN-6128), we will have multiple attempts in home SC but will always have only one for secondary as those are UAMs.

          • Nit: In TestUnmanagedApplicationManager::testForceKill, is there any specific exception message that you can assert as the second call can potential fail for other reasons.
          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for addressing my comments. I looked at the latest patch is pretty close, have a few comments: The pom changes for hadoop-yarn-server-common test dependencies are already committed as part of YARN-5411 , can you kindly rebase. Don't remove FederationProxyProviderUtil , we need it to manage Federation dependencies which currently has leaked into YarnServerSecurityUtils . Maybe AMRMProxyServiceUtils is a better place to put the createRMProxy but either ways the general utility class should not have any dependency to Federation. hreadpool is not a Yarn service, and no serviceStart call I don't get your comment as the threadpool is managed by UnmanagedAMPoolManager which is indeed a service: public class UnmanagedAMPoolManager extends AbstractService UnmanagedAMLauncher::submitAppAndGetAppId can be void. if (!appAttemptId.equals( this .attemptId)) { - I feel this check is not correct as once we have HA ( YARN-6128 ), we will have multiple attempts in home SC but will always have only one for secondary as those are UAMs. Nit: In TestUnmanagedApplicationManager::testForceKill , is there any specific exception message that you can assert as the second call can potential fail for other reasons.
          Hide
          botong Botong Huang added a comment -

          Thanks Subru Krishnan for the review! I've fixed all except:
          "this check is not correct as once we have HA (YARN-6128)". I will address this in YARN-6128 because we need to refactor the code there anyways.

          Good catch on "testForceKill", it turns out that the unit test is not written correctly and it was actually catching the Assert.fail() one line above and pass the test!

          Show
          botong Botong Huang added a comment - Thanks Subru Krishnan for the review! I've fixed all except: "this check is not correct as once we have HA ( YARN-6128 )". I will address this in YARN-6128 because we need to refactor the code there anyways. Good catch on "testForceKill", it turns out that the unit test is not written correctly and it was actually catching the Assert.fail() one line above and pass the test!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 44s Maven dependency ordering for branch
          +1 mvninstall 15m 7s YARN-2915 passed
          +1 compile 11m 18s YARN-2915 passed
          +1 checkstyle 0m 55s YARN-2915 passed
          +1 mvnsite 2m 36s YARN-2915 passed
          +1 mvneclipse 1m 31s YARN-2915 passed
          -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings.
          +1 javadoc 1m 52s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 58s the patch passed
          +1 compile 9m 59s the patch passed
          +1 javac 9m 59s the patch passed
          -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 48 unchanged - 1 fixed = 49 total (was 49)
          +1 mvnsite 2m 25s the patch passed
          +1 mvneclipse 1m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 32s the patch passed
          -1 javadoc 0m 23s hadoop-yarn-server-common in the patch failed.
          +1 unit 2m 31s hadoop-yarn-common in the patch passed.
          +1 unit 1m 23s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 15s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 39m 32s hadoop-yarn-server-resourcemanager in the patch passed.
          -1 asflicense 0m 34s The patch generated 1 ASF License warnings.
          128m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866717/YARN-5531-YARN-2915.v7.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9116e09c81ef 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / adbba7f
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15850/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15850/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 44s Maven dependency ordering for branch +1 mvninstall 15m 7s YARN-2915 passed +1 compile 11m 18s YARN-2915 passed +1 checkstyle 0m 55s YARN-2915 passed +1 mvnsite 2m 36s YARN-2915 passed +1 mvneclipse 1m 31s YARN-2915 passed -1 findbugs 1m 12s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. -1 findbugs 1m 11s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings. +1 javadoc 1m 52s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 9m 59s the patch passed +1 javac 9m 59s the patch passed -0 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 48 unchanged - 1 fixed = 49 total (was 49) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 1m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 32s the patch passed -1 javadoc 0m 23s hadoop-yarn-server-common in the patch failed. +1 unit 2m 31s hadoop-yarn-common in the patch passed. +1 unit 1m 23s hadoop-yarn-server-common in the patch passed. +1 unit 13m 15s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 39m 32s hadoop-yarn-server-resourcemanager in the patch passed. -1 asflicense 0m 34s The patch generated 1 ASF License warnings. 128m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866717/YARN-5531-YARN-2915.v7.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9116e09c81ef 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / adbba7f Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15850/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/15850/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15850/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 14m 26s YARN-2915 passed
          +1 compile 11m 49s YARN-2915 passed
          +1 checkstyle 0m 52s YARN-2915 passed
          +1 mvnsite 2m 25s YARN-2915 passed
          +1 mvneclipse 1m 30s YARN-2915 passed
          -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings.
          -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings.
          +1 javadoc 1m 53s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 10m 19s the patch passed
          +1 javac 10m 19s the patch passed
          +1 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49)
          +1 mvnsite 2m 29s the patch passed
          +1 mvneclipse 1m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 37s the patch passed
          +1 javadoc 1m 50s the patch passed
          +1 unit 2m 30s hadoop-yarn-common in the patch passed.
          +1 unit 1m 22s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 39m 29s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          127m 37s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ac17dc
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866735/YARN-5531-YARN-2915.v8.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6efe30f74960 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / adbba7f
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15851/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15851/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 14m 26s YARN-2915 passed +1 compile 11m 49s YARN-2915 passed +1 checkstyle 0m 52s YARN-2915 passed +1 mvnsite 2m 25s YARN-2915 passed +1 mvneclipse 1m 30s YARN-2915 passed -1 findbugs 1m 6s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 2 extant Findbugs warnings. -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager in YARN-2915 has 8 extant Findbugs warnings. +1 javadoc 1m 53s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 10m 19s the patch passed +1 javac 10m 19s the patch passed +1 checkstyle 0m 52s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49) +1 mvnsite 2m 29s the patch passed +1 mvneclipse 1m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 37s the patch passed +1 javadoc 1m 50s the patch passed +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 1m 22s hadoop-yarn-server-common in the patch passed. +1 unit 13m 7s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 39m 29s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 127m 37s Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866735/YARN-5531-YARN-2915.v8.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6efe30f74960 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / adbba7f Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15851/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15851/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15851/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for addressing my feedback. The latest patch (v8) LGTM. I only have a few nits:

          • Please do not log Token in YarnServerSecurityUtils::createRMProxy as it's not safe.
          • rmName is redundant in YarnServerSecurityUtils::createRMProxy as the RM_CLUSTER_ID is already in the conf.
          • Based on above comment, we don't need UnmanagedApplicationManager::createRMProxy.
          • Looks like you missed one of my comments:

            UnmanagedAMLauncher::submitAppAndGetAppId can be void.

          Karthik Kambatla, it'll be good if you take a quick look before I commit.

          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for addressing my feedback. The latest patch (v8) LGTM. I only have a few nits: Please do not log Token in YarnServerSecurityUtils::createRMProxy as it's not safe. rmName is redundant in YarnServerSecurityUtils::createRMProxy as the RM_CLUSTER_ID is already in the conf. Based on above comment, we don't need UnmanagedApplicationManager::createRMProxy . Looks like you missed one of my comments: UnmanagedAMLauncher::submitAppAndGetAppId can be void. Karthik Kambatla , it'll be good if you take a quick look before I commit.
          Hide
          botong Botong Huang added a comment -

          Thank Subru Krishnan for the review again. Please see v9 patch. UnmanagedApplicationManager::createRMProxy is still needed for unit test to override. All the rest is fixed.

          Show
          botong Botong Huang added a comment - Thank Subru Krishnan for the review again. Please see v9 patch. UnmanagedApplicationManager::createRMProxy is still needed for unit test to override. All the rest is fixed.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for the quick turnaround.

          Can you add a note in the Javadocs or annotate UnmanagedApplicationManager::createRMProxy that it's purely to enable tests.

          Nit: submitAppAndGetAppId can be renamed to simply submitUnmanagedApp.

          As discussed offline, please replace UnmanagedApplicationManager::setQueueName with passing queue name in UnmanagedAMPoolManager::createAndRegisterNewUAM.

          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for the quick turnaround. Can you add a note in the Javadocs or annotate UnmanagedApplicationManager::createRMProxy that it's purely to enable tests. Nit: submitAppAndGetAppId can be renamed to simply submitUnmanagedApp . As discussed offline, please replace UnmanagedApplicationManager::setQueueName with passing queue name in UnmanagedAMPoolManager::createAndRegisterNewUAM .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 50s Maven dependency ordering for branch
          +1 mvninstall 14m 35s YARN-2915 passed
          +1 compile 11m 33s YARN-2915 passed
          +1 checkstyle 1m 0s YARN-2915 passed
          +1 mvnsite 2m 34s YARN-2915 passed
          +1 mvneclipse 1m 32s YARN-2915 passed
          -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings.
          -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          +1 javadoc 1m 55s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 56s the patch passed
          +1 compile 10m 13s the patch passed
          +1 javac 10m 13s the patch passed
          +1 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49)
          +1 mvnsite 2m 32s the patch passed
          +1 mvneclipse 1m 28s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 48s the patch passed
          +1 javadoc 1m 47s the patch passed
          +1 unit 2m 26s hadoop-yarn-common in the patch passed.
          +1 unit 1m 19s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 2s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 40m 15s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          129m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867005/YARN-5531-YARN-2915.v9.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux fcaa113f2419 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 0a93a32
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15866/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15866/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15866/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15866/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 50s Maven dependency ordering for branch +1 mvninstall 14m 35s YARN-2915 passed +1 compile 11m 33s YARN-2915 passed +1 checkstyle 1m 0s YARN-2915 passed +1 mvnsite 2m 34s YARN-2915 passed +1 mvneclipse 1m 32s YARN-2915 passed -1 findbugs 1m 9s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings. -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. +1 javadoc 1m 55s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 56s the patch passed +1 compile 10m 13s the patch passed +1 javac 10m 13s the patch passed +1 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49) +1 mvnsite 2m 32s the patch passed +1 mvneclipse 1m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 48s the patch passed +1 javadoc 1m 47s the patch passed +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 1m 19s hadoop-yarn-server-common in the patch passed. +1 unit 13m 2s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 40m 15s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 129m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867005/YARN-5531-YARN-2915.v9.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux fcaa113f2419 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 0a93a32 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15866/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15866/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15866/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15866/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          botong Botong Huang added a comment -

          Thanks Subru Krishnan for the fast review. All comments addressed in v10 patch.

          Show
          botong Botong Huang added a comment - Thanks Subru Krishnan for the fast review. All comments addressed in v10 patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 48s Maven dependency ordering for branch
          +1 mvninstall 15m 3s YARN-2915 passed
          +1 compile 12m 17s YARN-2915 passed
          +1 checkstyle 0m 53s YARN-2915 passed
          +1 mvnsite 2m 31s YARN-2915 passed
          +1 mvneclipse 1m 30s YARN-2915 passed
          -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings.
          -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          +1 javadoc 1m 55s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 51s the patch passed
          +1 compile 9m 36s the patch passed
          +1 javac 9m 36s the patch passed
          -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 48 unchanged - 1 fixed = 49 total (was 49)
          +1 mvnsite 2m 23s the patch passed
          +1 mvneclipse 1m 26s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 18s the patch passed
          +1 javadoc 1m 48s the patch passed
          +1 unit 2m 26s hadoop-yarn-common in the patch passed.
          +1 unit 1m 19s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 2s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 40m 4s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          128m 53s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867138/YARN-5531-YARN-2915.v10.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9a1f97a44a62 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 0a93a32
          Default Java 1.8.0_121
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15873/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/15873/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 15m 3s YARN-2915 passed +1 compile 12m 17s YARN-2915 passed +1 checkstyle 0m 53s YARN-2915 passed +1 mvnsite 2m 31s YARN-2915 passed +1 mvneclipse 1m 30s YARN-2915 passed -1 findbugs 1m 10s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings. -1 findbugs 0m 50s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. +1 javadoc 1m 55s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 51s the patch passed +1 compile 9m 36s the patch passed +1 javac 9m 36s the patch passed -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 48 unchanged - 1 fixed = 49 total (was 49) +1 mvnsite 2m 23s the patch passed +1 mvneclipse 1m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 18s the patch passed +1 javadoc 1m 48s the patch passed +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 1m 19s hadoop-yarn-server-common in the patch passed. +1 unit 13m 2s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 40m 4s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 128m 53s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12867138/YARN-5531-YARN-2915.v10.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9a1f97a44a62 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 0a93a32 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/15873/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15873/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/15873/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for working on this, Botong Huang. I took a close look at the new files and skimmed through the remaining. Comments:

          1. Is yarn-server the best place for these? In the future, don't we want other clients to use this UAMPool? If we do change it to a different package, we need to think about the Visibility and Stability annotations.
          2. UnmanagedAMPoolManager:
            1. The create methods seem to be expect AppAttemptId from the user. Is that reasonable? Should it be the other way round, where we give the user AppAttemptId for the new app created?
            2. What are the benefits of using maps keyed by String, passed by the user. Why not just use ApplicationAttemptId? create methods could just return the app-attempt?
            3. Nit: In serviceStart, when creating maps, no need to specify the types starting Java 7.
            4. In serviceStart and serviceStop, shouldn't we call the equivalue super. methods right at the end? Otherwise, the state machine would transition the service to INITED or STOPPED even if it is not fully in that state?
            5. serviceStop
              1. I see the code tries to parallelize killing AMs. Is this necessary? How bad is sequential killing of apps?
              2. Nit: ExecutionCompletionService doesn't need the type in the creation.
              3. Why do we need the lock on the uamMap?
              4. Nit: Style choice. Where possible, I like to avoid nesting. The isEmpty check is for the logging. Can we not have the for nested.
              5. If we fail to kill the application, is catching the exception enough? Is there merit to retrying? Should we capture this state and throw an exception past this loop?
            6. createUAM should be annotated @VisibleForTesting
            7. Nit: allocateAsync: Don't see the need for variable uam.
            8. finishAM
              1. Nit: Don't see the need for variable uam.
              2. Don't we need to handle the case where the app is still registered? Retry?
          3. UnmanagedApplicationManager
            1. Should this class be called UnmanagedApplicationMaster?
            2. Constructor: Don't need to specify type when creating LinkedBlockingQueue
            3. UnmanagedAMLauncher
              1. It is not clear to me that this needs to be a separate inner class, outside of grouping methods that create an AM.
              2. submitAndGetAppId doesn't seem to really get app id?
              3. Why not use YarnClient? I understand this UAM pool is currently in yarn-server, but once we move this out, it should be easier.
              4. Would it be possible to have a single monitor method?
              5. Isn't one second too long a wait in monitor* methods?
            4. UnmanagedAMIdentifier can be private, so can be its methods.
            5. CallbackHandlerThread
              1. Can the combination of requestQueue and CallbackHandlerThread be achieved using a dispatcher?
              2. Should this thread be named HeartbeatHandlerThread or AMRequestHandlerThread? The thread is processing requests.
              3. We seem to throw RuntimeExceptions. Should these be YarnExceptions instead?
              4. Since the thread can crash, it is nicer to implement an UncaughtExceptionhandler for this thread?
            6. finishApplicationMaster
              1. Can the two if (rmProxy == null) checks be merged into one?
              2. Should the rmProxy.finishApplicationMaster be in a loop? Or, is one check and re-register enough?
            7. allocateAsync
              1. Is it okay to ignore the InterruptedException?
              2. The warning on UAM not being launched/registered seems unnecessary.
              3. Should the rmProxy == null && registerRequest == null check be first before we even queue this request?
          Show
          kasha Karthik Kambatla added a comment - Thanks for working on this, Botong Huang . I took a close look at the new files and skimmed through the remaining. Comments: Is yarn-server the best place for these? In the future, don't we want other clients to use this UAMPool? If we do change it to a different package, we need to think about the Visibility and Stability annotations. UnmanagedAMPoolManager: The create methods seem to be expect AppAttemptId from the user. Is that reasonable? Should it be the other way round, where we give the user AppAttemptId for the new app created? What are the benefits of using maps keyed by String, passed by the user. Why not just use ApplicationAttemptId? create methods could just return the app-attempt? Nit: In serviceStart, when creating maps, no need to specify the types starting Java 7. In serviceStart and serviceStop, shouldn't we call the equivalue super. methods right at the end? Otherwise, the state machine would transition the service to INITED or STOPPED even if it is not fully in that state? serviceStop I see the code tries to parallelize killing AMs. Is this necessary? How bad is sequential killing of apps? Nit: ExecutionCompletionService doesn't need the type in the creation. Why do we need the lock on the uamMap? Nit: Style choice. Where possible, I like to avoid nesting. The isEmpty check is for the logging. Can we not have the for nested. If we fail to kill the application, is catching the exception enough? Is there merit to retrying? Should we capture this state and throw an exception past this loop? createUAM should be annotated @VisibleForTesting Nit: allocateAsync: Don't see the need for variable uam. finishAM Nit: Don't see the need for variable uam. Don't we need to handle the case where the app is still registered? Retry? UnmanagedApplicationManager Should this class be called UnmanagedApplicationMaster? Constructor: Don't need to specify type when creating LinkedBlockingQueue UnmanagedAMLauncher It is not clear to me that this needs to be a separate inner class, outside of grouping methods that create an AM. submitAndGetAppId doesn't seem to really get app id? Why not use YarnClient? I understand this UAM pool is currently in yarn-server, but once we move this out, it should be easier. Would it be possible to have a single monitor method? Isn't one second too long a wait in monitor* methods? UnmanagedAMIdentifier can be private, so can be its methods. CallbackHandlerThread Can the combination of requestQueue and CallbackHandlerThread be achieved using a dispatcher? Should this thread be named HeartbeatHandlerThread or AMRequestHandlerThread? The thread is processing requests. We seem to throw RuntimeExceptions. Should these be YarnExceptions instead? Since the thread can crash, it is nicer to implement an UncaughtExceptionhandler for this thread? finishApplicationMaster Can the two if (rmProxy == null) checks be merged into one? Should the rmProxy.finishApplicationMaster be in a loop? Or, is one check and re-register enough? allocateAsync Is it okay to ignore the InterruptedException? The warning on UAM not being launched/registered seems unnecessary. Should the rmProxy == null && registerRequest == null check be first before we even queue this request?
          Hide
          botong Botong Huang added a comment - - edited

          Thanks Karthik Kambatla for the detailed comments! I have addressed most of them in v11 patch, the rest explanations are here:

          • 1 & 3.3.3. The reason we put it here is that Federation Interceptor (YARN-3666 and YARN-6511) in NM will be using UAM. Putting it in Yarn Client will result in cyclic dependencies for NM project.
          • 2.1-2 This is generalized from the Federation use case, where for one application we enforce the same applicationId in all sub-clusters (RMs in different sub-clusters use different epochs, so that their app Id won't overlap). uamID (sub-cluster ID really) is used to identify the UAMs. In v11 patch, I made the input attemptId becomes optional. If not supplied, the UAM will ask for an appID from RM first. In general, attempt id can be used as the uamID.
          • 2.5.1 Parallel kill is necessary for performance reason. In federation, the service stop of UAM pool is in the code path of Federation Interceptor shutdown, potentially blocking the application finish event in the NM where AM is running. Furthermore, when we try to kill the UAMs, RM in some sub-clusters might be failing over, which takes several minutes to come back. Sequential kill can be bad.
          • 2.5.5 Because of the above reason, I prefer not to retry here. One option is to throw the exception past this stop call, the user can handle the exception and retry if needed. In Federation Interceptor's case, we can simply catch it, log as warning and move on. What do you think?
          • 2.8.2 & 3.1 & 3.6.2 As mentioned with Subru Krishnan earlier, this UAM pool and UAM is more of a library for the actual UAM. The interface UAM pool expose to user is similar to ApplicationMasterProtocol (registerAM, allocate and finishAM), user is supposed to act like an AM and heartbeat to us. So for finishApplicationMaster, we abide by the protocol, if the UAM is still registered after the finishAM call, the user should retry.
          • 3.3.1 & 3.3.4 The launch UAM code is indeed a bit messy, I've cleaned up the code in v11. I merged the two monitor methods, might look a bit complex, can revert if needed.
          • 3.5.1 AsyncCallback works nicely in here. I think dispatcher can work as well, but I'd prefer to do that in another JIRA if needed.
          • 3.7.2-3 This is a corner use case for Federation. In federation interceptor, we handle the UAMs asynchronously. UAM is created the first time AM tries to ask for resource from certain sub-cluster. The register, allocate and finish calls for UAM are all triggered by heartbeats from AM. This means that all three calls are triggered asynchronously. For instance, while the register call for UAM is still pending (say because the UAM RM is falling over and the register call is blocked for five minutes), we need to allow the allocate calls to come in without exception and buffer them. Once the register succeeds later, we should be able to move on from there.
          Show
          botong Botong Huang added a comment - - edited Thanks Karthik Kambatla for the detailed comments! I have addressed most of them in v11 patch, the rest explanations are here: 1 & 3.3.3. The reason we put it here is that Federation Interceptor ( YARN-3666 and YARN-6511 ) in NM will be using UAM. Putting it in Yarn Client will result in cyclic dependencies for NM project. 2.1-2 This is generalized from the Federation use case, where for one application we enforce the same applicationId in all sub-clusters (RMs in different sub-clusters use different epochs, so that their app Id won't overlap). uamID (sub-cluster ID really) is used to identify the UAMs. In v11 patch, I made the input attemptId becomes optional. If not supplied, the UAM will ask for an appID from RM first. In general, attempt id can be used as the uamID. 2.5.1 Parallel kill is necessary for performance reason. In federation, the service stop of UAM pool is in the code path of Federation Interceptor shutdown, potentially blocking the application finish event in the NM where AM is running. Furthermore, when we try to kill the UAMs, RM in some sub-clusters might be failing over, which takes several minutes to come back. Sequential kill can be bad. 2.5.5 Because of the above reason, I prefer not to retry here. One option is to throw the exception past this stop call, the user can handle the exception and retry if needed. In Federation Interceptor's case, we can simply catch it, log as warning and move on. What do you think? 2.8.2 & 3.1 & 3.6.2 As mentioned with Subru Krishnan earlier, this UAM pool and UAM is more of a library for the actual UAM. The interface UAM pool expose to user is similar to ApplicationMasterProtocol (registerAM, allocate and finishAM), user is supposed to act like an AM and heartbeat to us. So for finishApplicationMaster , we abide by the protocol, if the UAM is still registered after the finishAM call, the user should retry. 3.3.1 & 3.3.4 The launch UAM code is indeed a bit messy, I've cleaned up the code in v11. I merged the two monitor methods, might look a bit complex, can revert if needed. 3.5.1 AsyncCallback works nicely in here. I think dispatcher can work as well, but I'd prefer to do that in another JIRA if needed. 3.7.2-3 This is a corner use case for Federation. In federation interceptor, we handle the UAMs asynchronously. UAM is created the first time AM tries to ask for resource from certain sub-cluster. The register, allocate and finish calls for UAM are all triggered by heartbeats from AM. This means that all three calls are triggered asynchronously. For instance, while the register call for UAM is still pending (say because the UAM RM is falling over and the register call is blocked for five minutes), we need to allow the allocate calls to come in without exception and buffer them. Once the register succeeds later, we should be able to move on from there.
          Hide
          kasha Karthik Kambatla added a comment -

          Sorry for the delay on this, Botong. Will take a look today if nothing urgent comes up at work.

          Show
          kasha Karthik Kambatla added a comment - Sorry for the delay on this, Botong. Will take a look today if nothing urgent comes up at work.
          Hide
          botong Botong Huang added a comment - - edited

          v12 patch: reset UAM responseId after re-register

          Show
          botong Botong Huang added a comment - - edited v12 patch: reset UAM responseId after re-register
          Hide
          kasha Karthik Kambatla added a comment -

          Botong Huang - apologies for not getting to the review. I don't think I ll be able to get it until Friday.

          Subru Krishnan, Carlo Curino - if you are comfortable with the patch, please go ahead and commit it. I ll try and circle back when I can and capture any pending work that is necessary.

          Show
          kasha Karthik Kambatla added a comment - Botong Huang - apologies for not getting to the review. I don't think I ll be able to get it until Friday. Subru Krishnan , Carlo Curino - if you are comfortable with the patch, please go ahead and commit it. I ll try and circle back when I can and capture any pending work that is necessary.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 57s Maven dependency ordering for branch
          +1 mvninstall 17m 37s YARN-2915 passed
          +1 compile 9m 58s YARN-2915 passed
          +1 checkstyle 1m 8s YARN-2915 passed
          +1 mvnsite 2m 43s YARN-2915 passed
          +1 mvneclipse 1m 36s YARN-2915 passed
          -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings.
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          +1 javadoc 1m 55s YARN-2915 passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 50s the patch passed
          +1 compile 7m 54s the patch passed
          +1 javac 7m 54s the patch passed
          -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 49 unchanged - 1 fixed = 53 total (was 50)
          +1 mvnsite 2m 24s the patch passed
          +1 mvneclipse 1m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          -1 javadoc 0m 25s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 162 unchanged - 0 fixed = 163 total (was 162)
          +1 unit 2m 33s hadoop-yarn-common in the patch passed.
          +1 unit 1m 24s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 59s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 45m 15s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          134m 58s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
            Nullcheck of attemptReport at line 433 of value previously dereferenced in org.apache.hadoop.yarn.server.uam.UnmanagedApplicationManager.monitorCurrentAppAttempt(ApplicationId, Set, YarnApplicationAttemptState) At UnmanagedApplicationManager.java:433 of value previously dereferenced in org.apache.hadoop.yarn.server.uam.UnmanagedApplicationManager.monitorCurrentAppAttempt(ApplicationId, Set, YarnApplicationAttemptState) At UnmanagedApplicationManager.java:[line 433]
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
            org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869708/YARN-5531-YARN-2915.v12.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 361896ed45d0 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / e47aba7
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16009/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16009/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 57s Maven dependency ordering for branch +1 mvninstall 17m 37s YARN-2915 passed +1 compile 9m 58s YARN-2915 passed +1 checkstyle 1m 8s YARN-2915 passed +1 mvnsite 2m 43s YARN-2915 passed +1 mvneclipse 1m 36s YARN-2915 passed -1 findbugs 1m 7s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings. -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. +1 javadoc 1m 55s YARN-2915 passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 50s the patch passed +1 compile 7m 54s the patch passed +1 javac 7m 54s the patch passed -0 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 49 unchanged - 1 fixed = 53 total (was 50) +1 mvnsite 2m 24s the patch passed +1 mvneclipse 1m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 4s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) -1 javadoc 0m 25s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 162 unchanged - 0 fixed = 163 total (was 162) +1 unit 2m 33s hadoop-yarn-common in the patch passed. +1 unit 1m 24s hadoop-yarn-server-common in the patch passed. +1 unit 13m 59s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 45m 15s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 134m 58s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common   Nullcheck of attemptReport at line 433 of value previously dereferenced in org.apache.hadoop.yarn.server.uam.UnmanagedApplicationManager.monitorCurrentAppAttempt(ApplicationId, Set, YarnApplicationAttemptState) At UnmanagedApplicationManager.java:433 of value previously dereferenced in org.apache.hadoop.yarn.server.uam.UnmanagedApplicationManager.monitorCurrentAppAttempt(ApplicationId, Set, YarnApplicationAttemptState) At UnmanagedApplicationManager.java: [line 433] Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869708/YARN-5531-YARN-2915.v12.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 361896ed45d0 3.13.0-108-generic #155-Ubuntu SMP Wed Jan 11 16:58:52 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / e47aba7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16009/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16009/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16009/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Karthik Kambatla for your thoughtful comments. I walked through them in detail with Botong Huang based on v12 patch. I feel he has addressed most of the possible ones excepts for:

          • Add Visibility and Stability annotations for the pool class.
          • Have a helper method in UnmanagedAMPoolManager that doesn't require the user to provide uamId. It'll will implicitly use the attempt id.
          • Move super. methods right at the end of serviceStart and serviceStop.
          • TODO for making shutdown non-blocking.
          • Javadocs to explain why UnmanagedAMIdentifier is public.
          • Consider moving rmProxy.finishApplicationMaster to be in a loop.
          • Change LOG levels in allocateAsync.
          • Nit: I prefer AMRequestHandlerThread.
          Show
          subru Subru Krishnan added a comment - Thanks Karthik Kambatla for your thoughtful comments. I walked through them in detail with Botong Huang based on v12 patch. I feel he has addressed most of the possible ones excepts for: Add Visibility and Stability annotations for the pool class. Have a helper method in UnmanagedAMPoolManager that doesn't require the user to provide uamId . It'll will implicitly use the attempt id. Move super. methods right at the end of serviceStart and serviceStop. TODO for making shutdown non-blocking. Javadocs to explain why UnmanagedAMIdentifier is public. Consider moving rmProxy.finishApplicationMaster to be in a loop. Change LOG levels in allocateAsync. Nit: I prefer AMRequestHandlerThread.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 52s Maven dependency ordering for branch
          +1 mvninstall 15m 15s YARN-2915 passed
          +1 compile 8m 18s YARN-2915 passed
          +1 checkstyle 0m 53s YARN-2915 passed
          +1 mvnsite 2m 27s YARN-2915 passed
          +1 mvneclipse 1m 29s YARN-2915 passed
          -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings.
          -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          +1 javadoc 1m 52s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 7m 45s the patch passed
          +1 javac 7m 45s the patch passed
          +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50)
          +1 mvnsite 2m 25s the patch passed
          +1 mvneclipse 1m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 35s the patch passed
          -1 javadoc 0m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 162 unchanged - 0 fixed = 163 total (was 162)
          +1 unit 2m 26s hadoop-yarn-common in the patch passed.
          +1 unit 1m 15s hadoop-yarn-server-common in the patch passed.
          +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 38m 56s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 35s The patch does not generate ASF License warnings.
          122m 57s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869947/YARN-5531-YARN-2915.v13.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 78cd9e834bd3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / e47aba7
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16023/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16023/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 52s Maven dependency ordering for branch +1 mvninstall 15m 15s YARN-2915 passed +1 compile 8m 18s YARN-2915 passed +1 checkstyle 0m 53s YARN-2915 passed +1 mvnsite 2m 27s YARN-2915 passed +1 mvneclipse 1m 29s YARN-2915 passed -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings. -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. +1 javadoc 1m 52s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 7m 45s the patch passed +1 javac 7m 45s the patch passed +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 49 unchanged - 1 fixed = 49 total (was 50) +1 mvnsite 2m 25s the patch passed +1 mvneclipse 1m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 35s the patch passed -1 javadoc 0m 22s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 162 unchanged - 0 fixed = 163 total (was 162) +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 1m 15s hadoop-yarn-server-common in the patch passed. +1 unit 12m 56s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 38m 56s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 35s The patch does not generate ASF License warnings. 122m 57s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.TestRMRestart Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869947/YARN-5531-YARN-2915.v13.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 78cd9e834bd3 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / e47aba7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16023/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16023/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16023/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 46s Maven dependency ordering for branch
          +1 mvninstall 17m 58s YARN-2915 passed
          +1 compile 9m 56s YARN-2915 passed
          +1 checkstyle 1m 1s YARN-2915 passed
          +1 mvnsite 2m 29s YARN-2915 passed
          +1 mvneclipse 1m 33s YARN-2915 passed
          -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings.
          -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings.
          +1 javadoc 2m 1s YARN-2915 passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 2m 4s the patch passed
          +1 compile 7m 57s the patch passed
          +1 javac 7m 57s the patch passed
          +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49)
          +1 mvnsite 2m 32s the patch passed
          +1 mvneclipse 1m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 5m 10s the patch passed
          +1 javadoc 1m 53s the patch passed
          +1 unit 2m 26s hadoop-yarn-common in the patch passed.
          +1 unit 1m 14s hadoop-yarn-server-common in the patch passed.
          +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed.
          -1 unit 43m 44s hadoop-yarn-server-resourcemanager in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          134m 24s



          Reason Tests
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869974/YARN-5531-YARN-2915.v14.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ea551506c183 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / e47aba7
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16026/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16026/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 46s Maven dependency ordering for branch +1 mvninstall 17m 58s YARN-2915 passed +1 compile 9m 56s YARN-2915 passed +1 checkstyle 1m 1s YARN-2915 passed +1 mvnsite 2m 29s YARN-2915 passed +1 mvneclipse 1m 33s YARN-2915 passed -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in YARN-2915 has 1 extant Findbugs warnings. -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in YARN-2915 has 5 extant Findbugs warnings. +1 javadoc 2m 1s YARN-2915 passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 2m 4s the patch passed +1 compile 7m 57s the patch passed +1 javac 7m 57s the patch passed +1 checkstyle 0m 53s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 48 unchanged - 1 fixed = 48 total (was 49) +1 mvnsite 2m 32s the patch passed +1 mvneclipse 1m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 5m 10s the patch passed +1 javadoc 1m 53s the patch passed +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 1m 14s hadoop-yarn-server-common in the patch passed. +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed. -1 unit 43m 44s hadoop-yarn-server-resourcemanager in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 134m 24s Reason Tests Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869974/YARN-5531-YARN-2915.v14.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ea551506c183 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / e47aba7 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16026/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16026/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16026/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for addressing the feedback. I have one minor comment - we should shutdown the AMRequestHandlerThread on forceKill too. It should be better to move it to an op inside the AMRequestHandlerThread.

          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for addressing the feedback. I have one minor comment - we should shutdown the AMRequestHandlerThread on forceKill too. It should be better to move it to an op inside the AMRequestHandlerThread .
          Hide
          botong Botong Huang added a comment -

          Good catch! I've fixed the thread shutdown in v15 patch. I've run test patch locally without issues.
          The test failure in v14 run (TestRMRestart.testFinishedAppRemovalAfterRMRestart) is not related to the patch, it is passing locally.

          Show
          botong Botong Huang added a comment - Good catch! I've fixed the thread shutdown in v15 patch. I've run test patch locally without issues. The test failure in v14 run (TestRMRestart.testFinishedAppRemovalAfterRMRestart) is not related to the patch, it is passing locally.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Botong Huang for the contribution! I just committed this to branch YARN-2915.

          Karthik Kambatla, I feel your feedback has been incorporated but will be good if you take a quick look next week or so before we merge the branch. Thanks.

          Show
          subru Subru Krishnan added a comment - Thanks Botong Huang for the contribution! I just committed this to branch YARN-2915 . Karthik Kambatla , I feel your feedback has been incorporated but will be good if you take a quick look next week or so before we merge the branch. Thanks.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 12s YARN-5531 does not apply to YARN-2915. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-5531
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870159/YARN-5531-YARN-2915.v15.patch
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16032/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 12s YARN-5531 does not apply to YARN-2915 . Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-5531 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12870159/YARN-5531-YARN-2915.v15.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/16032/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          botong Botong Huang added a comment -

          Thanks Subru Krishnan and Karthik Kambatla for the all the review and detailed feedback!

          Show
          botong Botong Huang added a comment - Thanks Subru Krishnan and Karthik Kambatla for the all the review and detailed feedback!

            People

            • Assignee:
              botong Botong Huang
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development