Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We introduce the notion of an RMProxy, running on each node (or once per rack). Upon start the AM is forced (via tokens and configuration) to direct all its requests to a new services running on the NM that provide a proxy to the central RM.

      This give us a place to:
      1) perform distributed scheduling decisions
      2) throttling mis-behaving AMs
      3) mask the access to a federation of RMs

      1. YARN-2884-V13.patch
        144 kB
        Kishore Chaliparambil
      2. YARN-2884-V12.patch
        144 kB
        Kishore Chaliparambil
      3. YARN-2884-V11.patch
        137 kB
        Kishore Chaliparambil
      4. YARN-2884-V10.patch
        138 kB
        Kishore Chaliparambil
      5. YARN-2884-V9.patch
        139 kB
        Kishore Chaliparambil
      6. YARN-2884-V8.patch
        136 kB
        Kishore Chaliparambil
      7. YARN-2884-V7.patch
        135 kB
        Kishore Chaliparambil
      8. YARN-2884-V6.patch
        134 kB
        Kishore Chaliparambil
      9. YARN-2884-V5.patch
        134 kB
        Kishore Chaliparambil
      10. YARN-2884-V4.patch
        134 kB
        Kishore Chaliparambil
      11. YARN-2884-V3.patch
        134 kB
        Kishore Chaliparambil
      12. YARN-2884-V2.patch
        134 kB
        Kishore Chaliparambil
      13. YARN-2884-V1.patch
        154 kB
        Kishore Chaliparambil

        Issue Links

          Activity

          Hide
          kishorch Kishore Chaliparambil added a comment -
          Show
          kishorch Kishore Chaliparambil added a comment - Thanks Subru Krishnan
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Thanks Jian!

          Show
          kishorch Kishore Chaliparambil added a comment - Thanks Jian!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2303 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2303/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2303 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2303/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #353 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/353/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #353 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/353/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #341 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/341/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #341 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/341/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2280/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2280 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2280/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1091 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1091/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1091 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1091/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #360 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/360/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #360 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/360/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          Hide
          subru Subru Krishnan added a comment -

          Thanks Kishore Chaliparambil for working on this and Jian He for reviewing/committing.

          Show
          subru Subru Krishnan added a comment - Thanks Kishore Chaliparambil for working on this and Jian He for reviewing/committing.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8412 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8412/)
          YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8412 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8412/ ) YARN-2884 . Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil (jianhe: rev 6f72f1e6003ab11679bebeb96f27f1f62b3b3e02) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/BaseAMRMProxyTest.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AbstractRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/test/java/org/apache/hadoop/yarn/conf/TestYarnConfigurationFields.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContext.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/PassThroughRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyTokenSecretManager.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/DefaultRequestInterceptor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/MockResourceManagerFacade.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/YarnServerSecurityUtils.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/RequestInterceptor.java
          Hide
          jianhe Jian He added a comment -

          Committed to trunk and branch-2,

          thanks Kishore Chaliparambil and Subru Krishnan !

          Show
          jianhe Jian He added a comment - Committed to trunk and branch-2, thanks Kishore Chaliparambil and Subru Krishnan !
          Hide
          kishorch Kishore Chaliparambil added a comment -

          The test failure is not related to the patch.

          Also could not address couple of issues:
          1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines.
          2) Checkstyle: Missing package-info.java file

          Show
          kishorch Kishore Chaliparambil added a comment - The test failure is not related to the patch. Also could not address couple of issues: 1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines. 2) Checkstyle: Missing package-info.java file
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 59s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 15s There were no new javadoc warning messages.
          +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 20s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          -1 checkstyle 3m 4s The applied patch generated 1 new checkstyle issues (total was 0, now 1).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 7m 0s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 7m 56s Tests passed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 58m 24s Tests failed in hadoop-yarn-server-resourcemanager.
              121m 35s  



          Reason Tests
          Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754282/YARN-2884-V13.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / bcc85e3
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9012/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9012/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 59s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 15s There were no new javadoc warning messages. +1 release audit 0m 26s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 20s The applied patch generated 1 new checkstyle issues (total was 211, now 211). -1 checkstyle 3m 4s The applied patch generated 1 new checkstyle issues (total was 0, now 1). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 7m 0s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 2s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 7m 56s Tests passed in hadoop-yarn-server-nodemanager. -1 yarn tests 58m 24s Tests failed in hadoop-yarn-server-resourcemanager.     121m 35s   Reason Tests Timed out tests org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754282/YARN-2884-V13.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / bcc85e3 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9012/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9012/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9012/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Fixed the one findbug issue that was found in the patch 12.

          Show
          kishorch Kishore Chaliparambil added a comment - Fixed the one findbug issue that was found in the patch 12.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          I investigated the Findbug and test failures.
          The test failures seem to be transient and does not happen on local builds. Also the Findbug report is empty and has no information.

          Show
          kishorch Kishore Chaliparambil added a comment - I investigated the Findbug and test failures. The test failures seem to be transient and does not happen on local builds. Also the Findbug report is empty and has no information.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 24s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 4s There were no new javac warning messages.
          +1 javadoc 10m 16s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 35s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 7m 4s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 6m 24s Tests failed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 0m 19s Tests failed in hadoop-yarn-server-resourcemanager.
              62m 32s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-nodemanager
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager
            hadoop.yarn.server.nodemanager.TestNodeManagerReboot
            hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor
            hadoop.yarn.server.nodemanager.TestNodeManagerResync
            hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
            hadoop.yarn.server.nodemanager.amrmproxy.TestAMRMProxyService
            hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch
            hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery
            hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
            hadoop.yarn.server.nodemanager.TestNodeManagerShutdown
          Failed build hadoop-yarn-server-resourcemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754145/YARN-2884-V12.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c83d13c
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9006/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9006/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 24s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 4s There were no new javac warning messages. +1 javadoc 10m 16s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 35s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 7m 4s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 24s Tests failed in hadoop-yarn-server-nodemanager. -1 yarn tests 0m 19s Tests failed in hadoop-yarn-server-resourcemanager.     62m 32s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Failed unit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager   hadoop.yarn.server.nodemanager.TestNodeManagerReboot   hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor   hadoop.yarn.server.nodemanager.TestNodeManagerResync   hadoop.yarn.server.nodemanager.TestNodeStatusUpdater   hadoop.yarn.server.nodemanager.amrmproxy.TestAMRMProxyService   hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch   hadoop.yarn.server.nodemanager.containermanager.TestContainerManagerRecovery   hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService   hadoop.yarn.server.nodemanager.TestNodeManagerShutdown Failed build hadoop-yarn-server-resourcemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754145/YARN-2884-V12.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c83d13c Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/9006/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9006/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9006/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          This patch fixes the issue with rolling Tokens in the proxy service. I tested the rolling tokens feature by setting the time interval to a smaller value and submitting long running job as a different user. The jobs finished successfully.

          The unused methods from the AMRMProxyTokenSecretManager has been removed too.

          I will create a new JIRA for adding test cases with some simulations for clusters and proxy.

          Show
          kishorch Kishore Chaliparambil added a comment - This patch fixes the issue with rolling Tokens in the proxy service. I tested the rolling tokens feature by setting the time interval to a smaller value and submitting long running job as a different user. The jobs finished successfully. The unused methods from the AMRMProxyTokenSecretManager has been removed too. I will create a new JIRA for adding test cases with some simulations for clusters and proxy.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Thanks Jian He. I will address these comments and upload the patch. Also as you suggested, I think I will create a new Jira for simulating the token renewal behavior in the proxy service since it might take more time.

          Show
          kishorch Kishore Chaliparambil added a comment - Thanks Jian He . I will address these comments and upload the patch. Also as you suggested, I think I will create a new Jira for simulating the token renewal behavior in the proxy service since it might take more time.
          Hide
          jianhe Jian He added a comment -

          Looks good to me overall, I think there are still some problems with the AMRMProxyToken implementation. Basically, long running service may not work with the AMRMProxy.

          1) below code in DefaultRequestInterceptor should create and return a new AMRMProxyToken in the final returned allocate response when needed. Otherwise, AM will fail to talk with AMRMTokenProxy after the key is rolled over in the AMRMTokenProxySecretManager.

            @Override
            public AllocateResponse allocate(AllocateRequest request)
                throws YarnException, IOException {
              if (LOG.isDebugEnabled()) {
                LOG.debug("Forwarding allocate request to the real YARN RM");
              }
              AllocateResponse allocateResponse = rmClient.allocate(request);
              if (allocateResponse.getAMRMToken() != null) {
                updateAMRMToken(allocateResponse.getAMRMToken());
              }
              return allocateResponse; <====
            }
          

          Below code in ApplicationMasterService#allocate shows how that is done.

                if (nextMasterKey != null
                    && nextMasterKey.getMasterKey().getKeyId() != amrmTokenIdentifier
                      .getKeyId()) {
                  RMAppAttemptImpl appAttemptImpl = (RMAppAttemptImpl)appAttempt;
                  Token<AMRMTokenIdentifier> amrmToken = appAttempt.getAMRMToken();
                  if (nextMasterKey.getMasterKey().getKeyId() !=
                      appAttemptImpl.getAMRMTokenKeyId()) {
                    LOG.info("The AMRMToken has been rolled-over. Send new AMRMToken back"
                        + " to application: " + applicationId);
                    amrmToken = rmContext.getAMRMTokenSecretManager()
                        .createAndGetAMRMToken(appAttemptId);
                    appAttemptImpl.setAMRMToken(amrmToken);
                  }
                  allocateResponse.setAMRMToken(org.apache.hadoop.yarn.api.records.Token
                    .newInstance(amrmToken.getIdentifier(), amrmToken.getKind()
                      .toString(), amrmToken.getPassword(), amrmToken.getService()
                      .toString()));
                }
          

          2) Some methods inside the AMRMProxyTokenSecretManager are not used at all. we may remove them ?

          3) I think we need at least 1 end-to-end test for this. We can use MiniYarnCluster to simulate the whole thing. AM talks with AMRMProxy which talks with RM to register/allocate/finish. In the test, we should also reduce the RM_AMRM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS so that we can simulate the token renew behavior. I'm ok to have a separate jira to track the end-to-end test, as this is a bit of work.

          Show
          jianhe Jian He added a comment - Looks good to me overall, I think there are still some problems with the AMRMProxyToken implementation. Basically, long running service may not work with the AMRMProxy. 1) below code in DefaultRequestInterceptor should create and return a new AMRMProxyToken in the final returned allocate response when needed. Otherwise, AM will fail to talk with AMRMTokenProxy after the key is rolled over in the AMRMTokenProxySecretManager. @Override public AllocateResponse allocate(AllocateRequest request) throws YarnException, IOException { if (LOG.isDebugEnabled()) { LOG.debug( "Forwarding allocate request to the real YARN RM" ); } AllocateResponse allocateResponse = rmClient.allocate(request); if (allocateResponse.getAMRMToken() != null ) { updateAMRMToken(allocateResponse.getAMRMToken()); } return allocateResponse; <==== } Below code in ApplicationMasterService#allocate shows how that is done. if (nextMasterKey != null && nextMasterKey.getMasterKey().getKeyId() != amrmTokenIdentifier .getKeyId()) { RMAppAttemptImpl appAttemptImpl = (RMAppAttemptImpl)appAttempt; Token<AMRMTokenIdentifier> amrmToken = appAttempt.getAMRMToken(); if (nextMasterKey.getMasterKey().getKeyId() != appAttemptImpl.getAMRMTokenKeyId()) { LOG.info( "The AMRMToken has been rolled-over. Send new AMRMToken back" + " to application: " + applicationId); amrmToken = rmContext.getAMRMTokenSecretManager() .createAndGetAMRMToken(appAttemptId); appAttemptImpl.setAMRMToken(amrmToken); } allocateResponse.setAMRMToken(org.apache.hadoop.yarn.api.records.Token .newInstance(amrmToken.getIdentifier(), amrmToken.getKind() .toString(), amrmToken.getPassword(), amrmToken.getService() .toString())); } 2) Some methods inside the AMRMProxyTokenSecretManager are not used at all. we may remove them ? 3) I think we need at least 1 end-to-end test for this. We can use MiniYarnCluster to simulate the whole thing. AM talks with AMRMProxy which talks with RM to register/allocate/finish. In the test, we should also reduce the RM_AMRM_TOKEN_MASTER_KEY_ROLLING_INTERVAL_SECS so that we can simulate the token renew behavior. I'm ok to have a separate jira to track the end-to-end test, as this is a bit of work.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 18s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 55s There were no new javac warning messages.
          +1 javadoc 10m 2s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 211, now 211).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 6m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 7m 44s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 53m 29s Tests passed in hadoop-yarn-server-resourcemanager.
              116m 18s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752399/YARN-2884-V11.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a4d9acc
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8913/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8913/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 18s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 55s There were no new javac warning messages. +1 javadoc 10m 2s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 211, now 211). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 6m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 7m 44s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 53m 29s Tests passed in hadoop-yarn-server-resourcemanager.     116m 18s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752399/YARN-2884-V11.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8913/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8913/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8913/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Removed the ApplicationConstants.java file from the patch because it is not required.

          Show
          kishorch Kishore Chaliparambil added a comment - Removed the ApplicationConstants.java file from the patch because it is not required.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 22m 8s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 49s There were no new javac warning messages.
          +1 javadoc 11m 44s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 3m 1s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 51s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 7m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 7m 32s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 56m 17s Tests passed in hadoop-yarn-server-resourcemanager.
              124m 2s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752377/YARN-2884-V10.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a4d9acc
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8912/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8912/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 22m 8s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 49s There were no new javac warning messages. +1 javadoc 11m 44s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 3m 1s The applied patch generated 1 new checkstyle issues (total was 237, now 237). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 51s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 7m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 59s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 7m 32s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 56m 17s Tests passed in hadoop-yarn-server-resourcemanager.     124m 2s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752377/YARN-2884-V10.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8912/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8912/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8912/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Uploaded YARN-2884-V10.patch. The changes in ContainerLaunch has been removed.

          Show
          kishorch Kishore Chaliparambil added a comment - Uploaded YARN-2884 -V10.patch. The changes in ContainerLaunch has been removed.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He, have created YARN-4083.

          Show
          subru Subru Krishnan added a comment - Thanks Jian He , have created YARN-4083 .
          Hide
          jianhe Jian He added a comment -

          To make this move faster, I think we can have a separate jira to address the scheduler address discovery problem. At least, MR job can run without the change.

          Show
          jianhe Jian He added a comment - To make this move faster, I think we can have a separate jira to address the scheduler address discovery problem. At least, MR job can run without the change.
          Hide
          subru Subru Krishnan added a comment -

          Just to add more context based on the offline discussions with @jian he, we can add a YARN_SCHEDULER_ADDRESS environment based scheduler discovery in the AMRMClient as an immediate first step. This will not cover all the AMs as AMRMClient is not used by custom AMs. Moreover apps can bring their own client JAR and the version can be older as long as it's backward compatible.

          Show
          subru Subru Krishnan added a comment - Just to add more context based on the offline discussions with @jian he, we can add a YARN_SCHEDULER_ADDRESS environment based scheduler discovery in the AMRMClient as an immediate first step. This will not cover all the AMs as AMRMClient is not used by custom AMs. Moreover apps can bring their own client JAR and the version can be older as long as it's backward compatible.
          Hide
          subru Subru Krishnan added a comment -

          Vinod Kumar Vavilapalli, thanks for your feedback. Let me first reiterate what I said to Jason Lowe's similar observation, I agree not only that we should move towards a better scheduler discovery model but completely decouple apps from platform configs. The reason we didn't go down the path you have suggested is it puts a dependency on updating all the AMs (which we don't own unlike Timeline service) to use the new discovery mechanism. The current approach though non-ideal is agnostic to AM. To force the AMs to do just that, we should prevent access to the NM's config. If all of you are OK with the consequence, I can go ahead and make the change.

          I think it'll be better if we open a separate JIRA to address the decoupling of app & platform config with an initial sub-task to handle scheduler discovery through environment as you suggested? In that case, we'll update the patch to remove the changes in ContainerLaunch that overrides the HADOOP_CONF_DIR and AFAIK, Jian He is OK with rest of the patch which he can commit asap. This will unblock us to use AMRMProxy with at least self contained apps like MapReduce, Spark which is our major workload.

          Show
          subru Subru Krishnan added a comment - Vinod Kumar Vavilapalli , thanks for your feedback. Let me first reiterate what I said to Jason Lowe 's similar observation, I agree not only that we should move towards a better scheduler discovery model but completely decouple apps from platform configs. The reason we didn't go down the path you have suggested is it puts a dependency on updating all the AMs (which we don't own unlike Timeline service) to use the new discovery mechanism. The current approach though non-ideal is agnostic to AM. To force the AMs to do just that, we should prevent access to the NM's config. If all of you are OK with the consequence, I can go ahead and make the change. I think it'll be better if we open a separate JIRA to address the decoupling of app & platform config with an initial sub-task to handle scheduler discovery through environment as you suggested? In that case, we'll update the patch to remove the changes in ContainerLaunch that overrides the HADOOP_CONF_DIR and AFAIK, Jian He is OK with rest of the patch which he can commit asap. This will unblock us to use AMRMProxy with at least self contained apps like MapReduce, Spark which is our major workload.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Jian He mentioned this offline and the configuration approach concerns me too.

          Stepping back, I think the current discovery of Scheduler by the apps is completely broken. Distributed Shell for e.g. works only because it is a java application and NM happens to put HADOOP_CONF_DIR in the classpath. Irrespective of this JIRA, we need to fix the scheduler discovery for the apps. The current way of depending on server configuration is unreliable in the face of rolling-upgrades.

          The specific solution in this JIRA further breaks rolling-upgrades and configuration updates. If and when, an admin forces client configuration changes, the config written by the Node will go out of sync. This overall makes the situation worse.

          I'd suggest that we start moving towards a better scheduler-discovery model. We have already done similar work with Timeline service (YARN-3039). We can implement part of that here - an environment based discovery - we can simply have an environment say YARN_SCHEDULER_ADDRESS for now set by the NodeManager into the AM-env, that is respected as the first level discovery mechanism. As we add more first-class discovery mechanisms, this env can take lesser precedence. This approach isn't too far from your current solution too, instead of pointing to a conf-dir env, you are pointing to a scheduler-address env directly.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Jian He mentioned this offline and the configuration approach concerns me too. Stepping back, I think the current discovery of Scheduler by the apps is completely broken. Distributed Shell for e.g. works only because it is a java application and NM happens to put HADOOP_CONF_DIR in the classpath. Irrespective of this JIRA, we need to fix the scheduler discovery for the apps. The current way of depending on server configuration is unreliable in the face of rolling-upgrades. The specific solution in this JIRA further breaks rolling-upgrades and configuration updates. If and when, an admin forces client configuration changes, the config written by the Node will go out of sync. This overall makes the situation worse. I'd suggest that we start moving towards a better scheduler-discovery model. We have already done similar work with Timeline service ( YARN-3039 ). We can implement part of that here - an environment based discovery - we can simply have an environment say YARN_SCHEDULER_ADDRESS for now set by the NodeManager into the AM-env, that is respected as the first level discovery mechanism. As we add more first-class discovery mechanisms, this env can take lesser precedence. This approach isn't too far from your current solution too, instead of pointing to a conf-dir env, you are pointing to a scheduler-address env directly.
          Hide
          subru Subru Krishnan added a comment -

          Jason Lowe, let me try to answer your question as this approach will not affect applications that ship their own configs. To run MapReduce in our cluster where AMRMProxy is enabled, the only change we made was to update resourcemanager.scheduler.address value to point to the amrmproxy.address. We thought this is acceptable as AMRMProxy (if enabled) is the Scheduler proxy for the apps and moreover quite easy to accomplish as we only had to update the MapReduce config only on our gateway machines from where MapReduce jobs are submitted. The rolling upgrade reliability as you rightly pointed out is maintained as MapReduce configs continues to be independent of node configs. FYI we also validated with Spark which exhibits the same characteristics.
          Ideally I agree that application configs should be decoupled from the server side configs for multiple reasons like rolling upgrades, security, etc but unfortunately many applications (REEF, Distributed Shell, etc) depend on the node configs today. So in summary the HADOOP_CONF_DIR modification will address applications that pick up configs from nodes without breaking self contained applications as the modified HADOOP_CONF_DIR does not show up on the latter's classpath.

          Show
          subru Subru Krishnan added a comment - Jason Lowe , let me try to answer your question as this approach will not affect applications that ship their own configs. To run MapReduce in our cluster where AMRMProxy is enabled, the only change we made was to update resourcemanager.scheduler.address value to point to the amrmproxy.address . We thought this is acceptable as AMRMProxy (if enabled) is the Scheduler proxy for the apps and moreover quite easy to accomplish as we only had to update the MapReduce config only on our gateway machines from where MapReduce jobs are submitted. The rolling upgrade reliability as you rightly pointed out is maintained as MapReduce configs continues to be independent of node configs. FYI we also validated with Spark which exhibits the same characteristics. Ideally I agree that application configs should be decoupled from the server side configs for multiple reasons like rolling upgrades, security, etc but unfortunately many applications (REEF, Distributed Shell, etc) depend on the node configs today. So in summary the HADOOP_CONF_DIR modification will address applications that pick up configs from nodes without breaking self contained applications as the modified HADOOP_CONF_DIR does not show up on the latter's classpath.
          Hide
          jlowe Jason Lowe added a comment -

          Note that not all applications pick up configs from the nodes, and I don't see how relying on a HADOOP_CONF_DIR modification will address them. For example, our setup runs a MapReduce job as a self-contained application – it does not reference the jars nor the configs on the cluster nodes. This makes rolling upgrades more reliable, otherwise a config change on the node could break old code in a job or new code in a job could break on an old node config. This happened in practice which is why our jobs no longer rely on confs from the nodes. HADOOP_CONF_DIR does not show up on the classpath for such applications, otherwise they would be relying on server-side configs and lead to the rolling upgrade instabilities.

          Any ideas on how to address the self-contained application scenario?

          Show
          jlowe Jason Lowe added a comment - Note that not all applications pick up configs from the nodes, and I don't see how relying on a HADOOP_CONF_DIR modification will address them. For example, our setup runs a MapReduce job as a self-contained application – it does not reference the jars nor the configs on the cluster nodes. This makes rolling upgrades more reliable, otherwise a config change on the node could break old code in a job or new code in a job could break on an old node config. This happened in practice which is why our jobs no longer rely on confs from the nodes. HADOOP_CONF_DIR does not show up on the classpath for such applications, otherwise they would be relying on server-side configs and lead to the rolling upgrade instabilities. Any ideas on how to address the self-contained application scenario?
          Hide
          subru Subru Krishnan added a comment -

          Jian He had an offline clarification - how does the AM know to connect to AMRMProxy and not the RM:
          If AMRMProxy is enabled, we need a HADOOP_CLIENT_CONF_DIR in every machine in the cluster which has a yarn-site with resourcemanager.scheduler.address pointing to the local AMRMProxy service and in NM ContainerLaunch we swap (only if AMRMProxy is enabled and HADOOP_CLIENT_CONF_DIR is not null to ensure full backward compatibility) the HADOOP_CONF_DIR in the AM container env to point to HADOOP_CLIENT_CONF_DIR. We tested with MapReduce, Spark & REEF and were able to run all of them successfully in a Federated YARN mode. Additionally this enhances the YARN security as currently server configs are leaked to all the AMs but with this change we can control every AMs view.

          Show
          subru Subru Krishnan added a comment - Jian He had an offline clarification - how does the AM know to connect to AMRMProxy and not the RM: If AMRMProxy is enabled, we need a HADOOP_CLIENT_CONF_DIR in every machine in the cluster which has a yarn-site with resourcemanager.scheduler.address pointing to the local AMRMProxy service and in NM ContainerLaunch we swap (only if AMRMProxy is enabled and HADOOP_CLIENT_CONF_DIR is not null to ensure full backward compatibility) the HADOOP_CONF_DIR in the AM container env to point to HADOOP_CLIENT_CONF_DIR. We tested with MapReduce, Spark & REEF and were able to run all of them successfully in a Federated YARN mode. Additionally this enhances the YARN security as currently server configs are leaked to all the AMs but with this change we can control every AMs view.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Tests failures in hadoop-yarn-common and hadoop-yarn-server-resourcemanager is not related to this patch. I confirmed that these tests work fine on my test environment

          Show
          kishorch Kishore Chaliparambil added a comment - Tests failures in hadoop-yarn-common and hadoop-yarn-server-resourcemanager is not related to this patch. I confirmed that these tests work fine on my test environment
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 27s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 48s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 29s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 6m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 57s Tests failed in hadoop-yarn-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 14s Tests passed in hadoop-yarn-server-nodemanager.
          -1 yarn tests 53m 11s Tests failed in hadoop-yarn-server-resourcemanager.
              114m 10s  



          Reason Tests
          Failed unit tests hadoop.yarn.util.TestRackResolver
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751175/YARN-2884-V9.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 30e342a
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8880/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8880/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 27s Pre-patch trunk has 7 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 48s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 237, now 237). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 29s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 6m 54s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 57s Tests failed in hadoop-yarn-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 14s Tests passed in hadoop-yarn-server-nodemanager. -1 yarn tests 53m 11s Tests failed in hadoop-yarn-server-resourcemanager.     114m 10s   Reason Tests Failed unit tests hadoop.yarn.util.TestRackResolver   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751175/YARN-2884-V9.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 30e342a Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8880/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8880/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8880/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Thanks Jian He for reviewing the patch. I have uploaded a new patch that addresses all your comments.

          Show
          kishorch Kishore Chaliparambil added a comment - Thanks Jian He for reviewing the patch. I have uploaded a new patch that addresses all your comments.
          Hide
          jianhe Jian He added a comment - - edited

          thanks Kishore Chaliparambil and Subru Krishnan, few comments on my side

          • why use containerTokenMasterKeyId for the AMRMTokenIdentifier keyId?
                int masterKeyId = containerTokenIdentifierForKey.getMasterKeyId();
                AMRMTokenIdentifier tokenId =
                    new AMRMTokenIdentifier(appAttemptId, masterKeyId);
            
          • I think we can catch the ApplicationEventType#FINISH_APPLICATION to catch the application finished signal ?
                this.dispatcher.register(ContainerEventType.class,
                    new ContainerEventHandler());
            
          • Today,the AMRMToken will be re-issued periodically, client on receiving a new token will updates its own ugi with the new token (see AMRMClientImpl#updateAMRMToken), the AMRMTokenProxyService may need to do the same if it gets a new token from RM.
          • AsynCallback.java file seems not used anywhere
          Show
          jianhe Jian He added a comment - - edited thanks Kishore Chaliparambil and Subru Krishnan , few comments on my side why use containerTokenMasterKeyId for the AMRMTokenIdentifier keyId? int masterKeyId = containerTokenIdentifierForKey.getMasterKeyId(); AMRMTokenIdentifier tokenId = new AMRMTokenIdentifier(appAttemptId, masterKeyId); I think we can catch the ApplicationEventType#FINISH_APPLICATION to catch the application finished signal ? this .dispatcher.register(ContainerEventType.class, new ContainerEventHandler()); Today,the AMRMToken will be re-issued periodically, client on receiving a new token will updates its own ugi with the new token (see AMRMClientImpl#updateAMRMToken), the AMRMTokenProxyService may need to do the same if it gets a new token from RM. AsynCallback.java file seems not used anywhere
          Hide
          subru Subru Krishnan added a comment -

          Thanks Kishore Chaliparambil for addressing all my comments. The latest patch LGTM.

          Jian He, can you please take a look.

          Show
          subru Subru Krishnan added a comment - Thanks Kishore Chaliparambil for addressing all my comments. The latest patch LGTM. Jian He , can you please take a look.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          The test failure (TestLogAggregationService.testLogAggregationServiceWithInterval) is unrelated to the patch. It looks like a transient failure. The same tests are working on local dev machine.

          Show
          kishorch Kishore Chaliparambil added a comment - The test failure (TestLogAggregationService.testLogAggregationServiceWithInterval) is unrelated to the patch. It looks like a transient failure. The same tests are working on local dev machine.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 58s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 2s There were no new javac warning messages.
          +1 javadoc 10m 11s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 7m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 53m 22s Tests passed in hadoop-yarn-server-resourcemanager.
              115m 43s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748587/YARN-2884-V8.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c3364ca
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8762/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8762/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 58s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 2s There were no new javac warning messages. +1 javadoc 10m 11s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 31s The applied patch generated 1 new checkstyle issues (total was 237, now 237). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 7m 4s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 0s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager. +1 yarn tests 53m 22s Tests passed in hadoop-yarn-server-resourcemanager.     115m 43s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748587/YARN-2884-V8.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c3364ca Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8762/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8762/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8762/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          The patch contains the following changes
          1. Use slf4j for logging.
          2. Renamed YarnServerUtils --> YarnServerSecurityUtils
          3. Added comments to public(& protected) methods

          Show
          kishorch Kishore Chaliparambil added a comment - The patch contains the following changes 1. Use slf4j for logging. 2. Renamed YarnServerUtils --> YarnServerSecurityUtils 3. Added comments to public(& protected) methods
          Hide
          subru Subru Krishnan added a comment -

          Thanks Kishore Chaliparambil for fixing the test patch warnings and addressing my feedback. The latest patch looks mostly good, some minor comments:

          • Since its a new service, can we use slf4j for logging.
          • Can we rename YarnServerUtils --> YarnServerSecurityUtils and update Javadocs accordingly as it has utility methods related to security only.
          • Can we reuse YarnServerUtils::selectAMRMTokenIdentifier or AMRMTokenSelector for AMRMProxyService::getFirstAMRMToken as they seem to be doing very similar functions.
          • A few public(& protected) methods have Javadoc missing, it'll be great if you add for those.
          Show
          subru Subru Krishnan added a comment - Thanks Kishore Chaliparambil for fixing the test patch warnings and addressing my feedback. The latest patch looks mostly good, some minor comments: Since its a new service, can we use slf4j for logging. Can we rename YarnServerUtils --> YarnServerSecurityUtils and update Javadocs accordingly as it has utility methods related to security only. Can we reuse YarnServerUtils::selectAMRMTokenIdentifier or AMRMTokenSelector for AMRMProxyService::getFirstAMRMToken as they seem to be doing very similar functions. A few public(& protected) methods have Javadoc missing, it'll be great if you add for those.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 19m 48s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 13s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 6m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 11s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 52m 26s Tests passed in hadoop-yarn-server-resourcemanager.
              111m 1s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748065/YARN-2884-V7.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 88d8736
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8723/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8723/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 48s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 13s The applied patch generated 1 new checkstyle issues (total was 237, now 237). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 6m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 11s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 52m 26s Tests passed in hadoop-yarn-server-resourcemanager.     111m 1s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748065/YARN-2884-V7.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 88d8736 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8723/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8723/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8723/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Fixed the javadoc warnings

          Show
          kishorch Kishore Chaliparambil added a comment - Fixed the javadoc warnings
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 7s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 25s There were no new javac warning messages.
          -1 javadoc 11m 29s The applied patch generated 4 additional warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 45s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 49s mvn install still works.
          +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse.
          +1 findbugs 8m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api.
          +1 yarn tests 2m 10s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 6m 33s Tests failed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 55m 5s Tests passed in hadoop-yarn-server-resourcemanager.
              121m 34s  



          Reason Tests
          Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747878/YARN-2884-V6.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c020b62
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/diffJavadocWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8711/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8711/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 7s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 25s There were no new javac warning messages. -1 javadoc 11m 29s The applied patch generated 4 additional warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 45s The applied patch generated 1 new checkstyle issues (total was 237, now 237). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 49s mvn install still works. +1 eclipse:eclipse 0m 42s The patch built with eclipse:eclipse. +1 findbugs 8m 30s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-api. +1 yarn tests 2m 10s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 26s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 33s Tests failed in hadoop-yarn-server-nodemanager. +1 yarn tests 55m 5s Tests passed in hadoop-yarn-server-resourcemanager.     121m 34s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747878/YARN-2884-V6.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c020b62 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/diffJavadocWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8711/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8711/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8711/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Fixed the checkstyle issues.

          Could not address couple of issues:
          1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines.
          2) Checkstyle:ApplicationMasterService.java: two lines are 81 char long
          3) Checkstyle:ContainerLaunch.java:689:3: Method length is 172 lines (max allowed is 150)
          4) Checkstyle: Warning that "member hides a field" is not fixed. This happens in even for all existing members
          5) Findbug report is empty

          Show
          kishorch Kishore Chaliparambil added a comment - Fixed the checkstyle issues. Could not address couple of issues: 1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines. 2) Checkstyle:ApplicationMasterService.java: two lines are 81 char long 3) Checkstyle:ContainerLaunch.java:689:3: Method length is 172 lines (max allowed is 150) 4) Checkstyle: Warning that "member hides a field" is not fixed. This happens in even for all existing members 5) Findbug report is empty
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 51s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 44s There were no new javac warning messages.
          +1 javadoc 9m 40s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 27s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          -1 checkstyle 3m 28s The applied patch generated 31 new checkstyle issues (total was 119, now 148).
          -1 checkstyle 3m 52s The applied patch generated 2 new checkstyle issues (total was 31, now 33).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 6m 45s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 52m 22s Tests passed in hadoop-yarn-server-resourcemanager.
              112m 33s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-nodemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747687/YARN-2884-V5.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 69b0957
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8702/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8702/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 51s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 44s There were no new javac warning messages. +1 javadoc 9m 40s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 27s The applied patch generated 1 new checkstyle issues (total was 237, now 237). -1 checkstyle 3m 28s The applied patch generated 31 new checkstyle issues (total was 119, now 148). -1 checkstyle 3m 52s The applied patch generated 2 new checkstyle issues (total was 31, now 33). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 6m 45s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 22s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 17s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 52m 22s Tests passed in hadoop-yarn-server-resourcemanager.     112m 33s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747687/YARN-2884-V5.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 69b0957 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8702/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8702/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8702/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Fixed issues reported by test-patch.

          The TestDeletionService test case failure seems to be a transient issue. It works fine on local dev. machine and also the patch did not change any code in that area.

          Show
          kishorch Kishore Chaliparambil added a comment - Fixed issues reported by test-patch. The TestDeletionService test case failure seems to be a transient issue. It works fine on local dev. machine and also the patch did not change any code in that area.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 20m 54s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 45s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 28s The applied patch generated 1 new checkstyle issues (total was 237, now 237).
          -1 checkstyle 3m 9s The applied patch generated 2 new checkstyle issues (total was 0, now 2).
          +1 whitespace 0m 2s The patch has no lines that end in whitespace.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          -1 findbugs 6m 49s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common.
          -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 52m 19s Tests passed in hadoop-yarn-server-resourcemanager.
              112m 12s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-nodemanager
          Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747612/YARN-2884-V4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 69b0957
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8697/testReport/
          Java 1.7.0_55
          uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8697/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 20m 54s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 45s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 28s The applied patch generated 1 new checkstyle issues (total was 237, now 237). -1 checkstyle 3m 9s The applied patch generated 2 new checkstyle issues (total was 0, now 2). +1 whitespace 0m 2s The patch has no lines that end in whitespace. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. -1 findbugs 6m 49s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 55s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 24s Tests passed in hadoop-yarn-server-common. -1 yarn tests 6m 7s Tests failed in hadoop-yarn-server-nodemanager. +1 yarn tests 52m 19s Tests passed in hadoop-yarn-server-resourcemanager.     112m 12s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747612/YARN-2884-V4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 69b0957 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8697/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8697/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8697/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Fixed the checkstyle issues.

          Could not address couple of issues:
          1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines.
          2) Findbug report is empty

          Show
          kishorch Kishore Chaliparambil added a comment - Fixed the checkstyle issues. Could not address couple of issues: 1) Checkstyle: YarnConfiguration.java - File length exceeds 2000 lines. 2) Findbug report is empty
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 21m 22s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 7m 43s There were no new javac warning messages.
          +1 javadoc 9m 53s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 25s The applied patch generated 2 new checkstyle issues (total was 237, now 238).
          -1 checkstyle 3m 11s The applied patch generated 2 new checkstyle issues (total was 0, now 2).
          -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          -1 findbugs 6m 51s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
          +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common.
          +1 yarn tests 6m 19s Tests passed in hadoop-yarn-server-nodemanager.
          +1 yarn tests 52m 22s Tests passed in hadoop-yarn-server-resourcemanager.
              113m 9s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-nodemanager



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747453/YARN-2884-V3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3e6fce9
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8686/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8686/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 21m 22s Pre-patch trunk has 6 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 7m 43s There were no new javac warning messages. +1 javadoc 9m 53s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 25s The applied patch generated 2 new checkstyle issues (total was 237, now 238). -1 checkstyle 3m 11s The applied patch generated 2 new checkstyle issues (total was 0, now 2). -1 whitespace 0m 2s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. -1 findbugs 6m 51s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. +1 yarn tests 1m 56s Tests passed in hadoop-yarn-common. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-server-common. +1 yarn tests 6m 19s Tests passed in hadoop-yarn-server-nodemanager. +1 yarn tests 52m 22s Tests passed in hadoop-yarn-server-resourcemanager.     113m 9s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747453/YARN-2884-V3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3e6fce9 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/trunkFindbugsWarningshadoop-yarn-server-common.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/diffcheckstylehadoop-yarn-server-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt hadoop-yarn-server-resourcemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8686/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8686/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8686/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Uploading a new version of the patch

          Show
          kishorch Kishore Chaliparambil added a comment - Uploading a new version of the patch
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12747414/YARN-2884-V2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 3e6fce9
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8684/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12747414/YARN-2884-V2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 3e6fce9 Console output https://builds.apache.org/job/PreCommit-YARN-Build/8684/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Attached a new patch that addresses review comments from Subru Krishnan

          Show
          kishorch Kishore Chaliparambil added a comment - Attached a new patch that addresses review comments from Subru Krishnan
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Thanks Subru for reviewing the patch. I will address these comments and also fix the test patch issues.

          Show
          kishorch Kishore Chaliparambil added a comment - Thanks Subru for reviewing the patch. I will address these comments and also fix the test patch issues.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Kishore Chaliparambil for posting the patch. I have a few comments:

          • The current implementation assumes that we will set the AMRMProxy address as the RM scheduler address in the client configuration. This will work for MapReduce, Spark etc where the client configuration is passed to the AM. But we need to explicitly override the RM scheduler address via the container launch environment to allow proxying more generically to work with all AMs like DistributedShell, REEF, etc
          • In AMRMProxyService, authorizeRequest is the exact same check as done by ApplicationMasterService so it'll be better to refactor the code to reuse for manageability.
          • Can we use the AMRMTokenSelector to select the AMRMToken in AMRMProxyService.
          • I see that the MasterKeyRoller is used in multiple places. We should have a RolloverSecretManager that does the rollover and have AMRMProxyTokenSecretManager (and others) extend it.

          There are few test patch issues in the first version of the patch, looks mostly to do with whitespaces. Can you take a look & fix those in the next iteration.

          Show
          subru Subru Krishnan added a comment - Thanks Kishore Chaliparambil for posting the patch. I have a few comments: The current implementation assumes that we will set the AMRMProxy address as the RM scheduler address in the client configuration. This will work for MapReduce , Spark etc where the client configuration is passed to the AM. But we need to explicitly override the RM scheduler address via the container launch environment to allow proxying more generically to work with all AMs like DistributedShell , REEF , etc In AMRMProxyService , authorizeRequest is the exact same check as done by ApplicationMasterService so it'll be better to refactor the code to reuse for manageability. Can we use the AMRMTokenSelector to select the AMRMToken in AMRMProxyService . I see that the MasterKeyRoller is used in multiple places. We should have a RolloverSecretManager that does the rollover and have AMRMProxyTokenSecretManager (and others) extend it. There are few test patch issues in the first version of the patch, looks mostly to do with whitespaces. Can you take a look & fix those in the next iteration.
          Hide
          subru Subru Krishnan added a comment -

          To give more context on the approach we took, please find below the summary of the offline discussions we had with Kishore Chaliparambil, Jian He,Wangda Tan, Zhijie Shen, Konstantinos Karanasos,Chris Douglas.

          One of the main drivers for the discussion was whether AMRMProxy service needs to be man-in-the-middle between RM and NM in order for sucessful SASL handshake. On investigation we realized that it was necessary for us to swap the AMRMToken as AM would register with the AMRMProxy service instead of the RM & we need to validate the AMRMToken. To achieve this we need either the RM’s secret key or generate & swap AMRMToken in the AMRMProxy and we went for the latter approach for obvious reasons.
          We considered a few options to plug in AMRMProxy to the NM:
          · Adding AMRMProxy as an auxiliary service: This looked the minimally invasive method but AMRMProxy requires access to NM state (SecretManager for generating local AMRMTokens, StateStore for persisting/recovering across NM restarts without killing the AM, etc). We want to isolate aux services from the NM and hence do not want to provide access to internal states.
          · Making the NM ContainerManager pluggable and implementing AMRMProxy as a custom ContainerManager that extends the default ContainerManagerImpl: This would give us all the leverage needed to implement the AMRMProxy, i.e. access to the NM context, ability to man-in-the-middle container lifecycle events, etc. But this would increase the complexity of the already heavy ContainerManager as we plan to support multiple handlers like Federation (YARN-2915), distributed scheduling (YARN-2877) in the AMRMProxy. Additionally we want to retain the flexibility of deploying AMRMProxy as an independent daemon in the future.

          So the final approach we decided was to plug in AMRMProxy as an independent first class service in the NM and have a flag to enable/disable it. We added an AM container pre-start hook in the ContainerManager where we swap the AMRMToken issued by the RM with one issued locally by the AMRMProxy. On receiving the register application call, AMRMProxy swaps back the original token issued by RM and forwards the request.

          Show
          subru Subru Krishnan added a comment - To give more context on the approach we took, please find below the summary of the offline discussions we had with Kishore Chaliparambil , Jian He , Wangda Tan , Zhijie Shen , Konstantinos Karanasos , Chris Douglas . One of the main drivers for the discussion was whether AMRMProxy service needs to be man-in-the-middle between RM and NM in order for sucessful SASL handshake. On investigation we realized that it was necessary for us to swap the AMRMToken as AM would register with the AMRMProxy service instead of the RM & we need to validate the AMRMToken. To achieve this we need either the RM’s secret key or generate & swap AMRMToken in the AMRMProxy and we went for the latter approach for obvious reasons. We considered a few options to plug in AMRMProxy to the NM: · Adding AMRMProxy as an auxiliary service: This looked the minimally invasive method but AMRMProxy requires access to NM state (SecretManager for generating local AMRMTokens, StateStore for persisting/recovering across NM restarts without killing the AM, etc). We want to isolate aux services from the NM and hence do not want to provide access to internal states. · Making the NM ContainerManager pluggable and implementing AMRMProxy as a custom ContainerManager that extends the default ContainerManagerImpl: This would give us all the leverage needed to implement the AMRMProxy, i.e. access to the NM context, ability to man-in-the-middle container lifecycle events, etc. But this would increase the complexity of the already heavy ContainerManager as we plan to support multiple handlers like Federation ( YARN-2915 ), distributed scheduling ( YARN-2877 ) in the AMRMProxy. Additionally we want to retain the flexibility of deploying AMRMProxy as an independent daemon in the future. So the final approach we decided was to plug in AMRMProxy as an independent first class service in the NM and have a flag to enable/disable it. We added an AM container pre-start hook in the ContainerManager where we swap the AMRMToken issued by the RM with one issued locally by the AMRMProxy. On receiving the register application call, AMRMProxy swaps back the original token issued by RM and forwards the request.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 24m 22s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 5 new or modified test files.
          +1 javac 8m 48s There were no new javac warning messages.
          -1 javadoc 9m 45s The applied patch generated 6 additional warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 2m 15s The applied patch generated 18 new checkstyle issues (total was 211, now 228).
          -1 checkstyle 2m 40s The applied patch generated 2 new checkstyle issues (total was 0, now 2).
          -1 whitespace 0m 7s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 22s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 4m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api.
          -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common.
          -1 yarn tests 6m 9s Tests failed in hadoop-yarn-server-nodemanager.
              61m 0s  



          Reason Tests
          FindBugs module:hadoop-yarn-server-nodemanager
          Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.nodemanager.TestDeletionService



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12746687/YARN-2884-V1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / ee98d63
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffJavadocWarnings.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/whitespace.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-api.txt
          hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-common.txt
          hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8630/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 24m 22s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 5 new or modified test files. +1 javac 8m 48s There were no new javac warning messages. -1 javadoc 9m 45s The applied patch generated 6 additional warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 2m 15s The applied patch generated 18 new checkstyle issues (total was 211, now 228). -1 checkstyle 2m 40s The applied patch generated 2 new checkstyle issues (total was 0, now 2). -1 whitespace 0m 7s The patch has 8 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 22s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 4m 21s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 yarn tests 0m 25s Tests passed in hadoop-yarn-api. -1 yarn tests 1m 54s Tests failed in hadoop-yarn-common. -1 yarn tests 6m 9s Tests failed in hadoop-yarn-server-nodemanager.     61m 0s   Reason Tests FindBugs module:hadoop-yarn-server-nodemanager Failed unit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746687/YARN-2884-V1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / ee98d63 javadoc https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffJavadocWarnings.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/whitespace.txt Findbugs warnings https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-common test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-common.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8630/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8630/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8630/console This message was automatically generated.
          Hide
          kishorch Kishore Chaliparambil added a comment -

          Added a new service in NodeManager called AMRMProxyService. This is by default disabled and can be enabled using "yarn.nodemanager.amrmproxy.enable". This service can be used for proxying calls from the Application Master to the Resource Manager.

          This feature is needed for:
          1) Federation - to transparently scale the application across multiple sub-clusters (YARN-2915)
          2) Mercury – to queue containers directly in NMs for distributed scheduling (YARN-2877)
          3) Security/Throttling etc. – prevent DDoS attacks on RM

          Show
          kishorch Kishore Chaliparambil added a comment - Added a new service in NodeManager called AMRMProxyService. This is by default disabled and can be enabled using "yarn.nodemanager.amrmproxy.enable". This service can be used for proxying calls from the Application Master to the Resource Manager. This feature is needed for: 1) Federation - to transparently scale the application across multiple sub-clusters ( YARN-2915 ) 2) Mercury – to queue containers directly in NMs for distributed scheduling ( YARN-2877 ) 3) Security/Throttling etc. – prevent DDoS attacks on RM
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Karthik Kambatla, Carlo Curino, Subru Krishnan, given that this proxy/agent will only focus on the AM-RM communication, we may also explicitly call it AMRMProxy or AMRMAgent (following the naming convention of the already existing AMRMClient* classes).

          Junping Du I just added a comment in the umbrella JIRA (YARN-2877), trying to give some more details.
          We are not proposing to substitute all scheduling decisions with distributed ones. The guaranteed-start containers will continue to be scheduled by the central RM. However, the queueable ones will be scheduled in a distributed fashion.
          The first candidate for queueable containers is the short-running tasks, in which the overhead of contacting the central RM is a significant part of the overall task execution time. Scheduling these requests without contacting the central RM will reduce their latency, increase the utilization of the cluster (no idle resources waiting to contact the RM), while it will offload the central RM (which is good for scaling in big clusters).

          Show
          kkaranasos Konstantinos Karanasos added a comment - Karthik Kambatla , Carlo Curino , Subru Krishnan , given that this proxy/agent will only focus on the AM-RM communication, we may also explicitly call it AMRMProxy or AMRMAgent (following the naming convention of the already existing AMRMClient* classes). Junping Du I just added a comment in the umbrella JIRA ( YARN-2877 ), trying to give some more details. We are not proposing to substitute all scheduling decisions with distributed ones. The guaranteed-start containers will continue to be scheduled by the central RM. However, the queueable ones will be scheduled in a distributed fashion. The first candidate for queueable containers is the short-running tasks, in which the overhead of contacting the central RM is a significant part of the overall task execution time. Scheduling these requests without contacting the central RM will reduce their latency, increase the utilization of the cluster (no idle resources waiting to contact the RM), while it will offload the central RM (which is good for scaling in big clusters).
          Hide
          djp Junping Du added a comment -

          I don't think the name matter too much ...
          IMO, this sounds like a complicated effort. Before we go ahead, may be we should have analysis on the motivation towards "distributed scheduling decisions"? - What we could gain there and what we could lost in potential?

          Show
          djp Junping Du added a comment - I don't think the name matter too much ... IMO, this sounds like a complicated effort. Before we go ahead, may be we should have analysis on the motivation towards "distributed scheduling decisions"? - What we could gain there and what we could lost in potential?
          Hide
          kasha Karthik Kambatla added a comment -

          RMAgent seems okay to me.

          RMProxy is responsible to create a Proxy depending on the protocol the "client" wants to converse with the RM.

          Show
          kasha Karthik Kambatla added a comment - RMAgent seems okay to me. RMProxy is responsible to create a Proxy depending on the protocol the "client" wants to converse with the RM.
          Hide
          kkaranasos Konstantinos Karanasos added a comment -

          Karthik, just a clarification: what is the current RMProxy responsible for?
          As Carlo says, the functionality needed for the distributed scheduling is explained in more detail in YARN-2885, where we introduce the LocalRM.

          Show
          kkaranasos Konstantinos Karanasos added a comment - Karthik, just a clarification: what is the current RMProxy responsible for? As Carlo says, the functionality needed for the distributed scheduling is explained in more detail in YARN-2885 , where we introduce the LocalRM.
          Hide
          subru Subru Krishnan added a comment -

          What about "RMAgent" ?

          Show
          subru Subru Krishnan added a comment - What about "RMAgent" ?
          Hide
          curino Carlo Curino added a comment -

          I agree we should give it another name... but the LocalRM is a slightly different concept YARN-2885, i.e., it is the logic making distributed scheduling decisions.
          The "Proxy" itself is just the mechanics to hijack the connection between AM-RM, which we will need for some more work on federating multiple RMs (JIRAs coming soon).

          Hence the need to call out separately the architectural piece (proxy) and the distributed scheduling logic (LocalRM). Any name suggestion?

          Show
          curino Carlo Curino added a comment - I agree we should give it another name... but the LocalRM is a slightly different concept YARN-2885 , i.e., it is the logic making distributed scheduling decisions. The "Proxy" itself is just the mechanics to hijack the connection between AM-RM, which we will need for some more work on federating multiple RMs (JIRAs coming soon). Hence the need to call out separately the architectural piece (proxy) and the distributed scheduling logic (LocalRM). Any name suggestion?
          Hide
          kasha Karthik Kambatla added a comment -

          Given we already have an RMProxy, can we go with LocalRM as Sriram suggested on YARN-2877?

          Show
          kasha Karthik Kambatla added a comment - Given we already have an RMProxy, can we go with LocalRM as Sriram suggested on YARN-2877 ?

            People

            • Assignee:
              kishorch Kishore Chaliparambil
              Reporter:
              curino Carlo Curino
            • Votes:
              0 Vote for this issue
              Watchers:
              27 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development