Details

    • Hadoop Flags:
      Reviewed

      Description

      This JIRA proposes creating a failover proxy for Federation based on the cluster membership information in the StateStore that can be used by both Router & AMRMProxy

      1. YARN-3673-YARN-2915-v1.patch
        36 kB
        Subru Krishnan
      2. YARN-3673-YARN-2915-v2.patch
        36 kB
        Subru Krishnan
      3. YARN-3673-YARN-2915-v3.patch
        37 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          First version of the FederationRMFailoverProxyProvider that uses the FederationStateStore to determine the ResourceManager to connect to.

          Show
          subru Subru Krishnan added a comment - First version of the FederationRMFailoverProxyProvider that uses the FederationStateStore to determine the ResourceManager to connect to.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 53s Maven dependency ordering for branch
          +1 mvninstall 6m 46s YARN-2915 passed
          +1 compile 2m 21s YARN-2915 passed
          +1 checkstyle 0m 40s YARN-2915 passed
          +1 mvnsite 1m 47s YARN-2915 passed
          +1 mvneclipse 0m 55s YARN-2915 passed
          +1 findbugs 3m 5s YARN-2915 passed
          +1 javadoc 1m 16s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 2m 30s the patch passed
          +1 javac 2m 30s the patch passed
          -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 235 unchanged - 0 fixed = 238 total (was 235)
          +1 mvnsite 1m 47s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 2s the patch passed
          -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 1 new + 4579 unchanged - 0 fixed = 4580 total (was 4579)
          -1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 4 new + 160 unchanged - 0 fixed = 164 total (was 160)
          +1 unit 0m 25s hadoop-yarn-api in the patch passed.
          +1 unit 2m 21s hadoop-yarn-common in the patch passed.
          +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
          -1 unit 16m 12s hadoop-yarn-client in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          51m 40s



          Reason Tests
          Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824279/YARN-3673-YARN-2915-v1.patch
          JIRA Issue YARN-3673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux b5ec09d34476 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 8abb0a8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12822/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12822/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 6m 46s YARN-2915 passed +1 compile 2m 21s YARN-2915 passed +1 checkstyle 0m 40s YARN-2915 passed +1 mvnsite 1m 47s YARN-2915 passed +1 mvneclipse 0m 55s YARN-2915 passed +1 findbugs 3m 5s YARN-2915 passed +1 javadoc 1m 16s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 2m 30s the patch passed +1 javac 2m 30s the patch passed -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 235 unchanged - 0 fixed = 238 total (was 235) +1 mvnsite 1m 47s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 2s the patch passed -1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 1 new + 4579 unchanged - 0 fixed = 4580 total (was 4579) -1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 4 new + 160 unchanged - 0 fixed = 164 total (was 160) +1 unit 0m 25s hadoop-yarn-api in the patch passed. +1 unit 2m 21s hadoop-yarn-common in the patch passed. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. -1 unit 16m 12s hadoop-yarn-client in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 51m 40s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824279/YARN-3673-YARN-2915-v1.patch JIRA Issue YARN-3673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux b5ec09d34476 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 8abb0a8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12822/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12822/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12822/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -
          • I think for this patch the FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flag are not needed. To use FederationRMFailoverProxyProvider, we just need to change the provider config to use the FederationRMFailoverProxyProvider class? Looks to me the FederationRMFailoverProxyProvider is just one way to retrieve the RM proxy object as compared to other existing ones.
          • Why do we need to store the tokens and then re-add the tokens?
            private void addOriginalTokens(UserGroupInformation currentUser) {
              if (originalTokens == null || originalTokens.isEmpty()) {
                return;
              }
              for (Token<? extends TokenIdentifier> token : originalTokens) {
                currentUser.addToken(token);
              }
            }
            
          • It appears to me the FederationStateStoreFacade is not needed for this patch. It is used in getProxyInternal(federationFailoverEnabled) only, while if federationFailoverEnabled is true (which should be always the case), the FederationStateStoreFacade will then always flushes the cache and request the state-store for the RMAddress.
            On the other hand, in the getProxy() method, we already have a current variable to cache the current object, which cause the cached object in the FederationStateStoreFacade essentially not used?
          Show
          jianhe Jian He added a comment - I think for this patch the FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flag are not needed. To use FederationRMFailoverProxyProvider, we just need to change the provider config to use the FederationRMFailoverProxyProvider class? Looks to me the FederationRMFailoverProxyProvider is just one way to retrieve the RM proxy object as compared to other existing ones. Why do we need to store the tokens and then re-add the tokens? private  void addOriginalTokens(UserGroupInformation currentUser) {   if  (originalTokens ==  null  || originalTokens.isEmpty()) {     return ;   }   for  (Token<?  extends  TokenIdentifier> token : originalTokens) {     currentUser.addToken(token);   } } It appears to me the FederationStateStoreFacade is not needed for this patch. It is used in getProxyInternal(federationFailoverEnabled) only, while if federationFailoverEnabled is true (which should be always the case), the FederationStateStoreFacade will then always flushes the cache and request the state-store for the RMAddress. On the other hand, in the getProxy() method, we already have a current variable to cache the current object, which cause the cached object in the FederationStateStoreFacade essentially not used?
          Hide
          jianhe Jian He added a comment -

          Test: IIUC, I don't actually see FederationRMFailoverProxyProvider configured in the test, is it tested?

          Show
          jianhe Jian He added a comment - Test: IIUC, I don't actually see FederationRMFailoverProxyProvider configured in the test, is it tested?
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for reviewing the patch and your detailed feedback.

          I think for this patch the FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flag are not needed

          There is a nuance here that unlike the other failover providers which are only initialized when RM HA is enabled,FederationRMFailoverProxyProvider is used with both standalone RM and RM HA as the main purpose is to bypass yarn-site and obtain the RM connection address from FederationStateStore.
          The FEDERATION_ENABLED flag indicates that irrespective of whether RM has HA enabled or not, we should configure the FederationRMFailoverProxyProvider.
          When we have both Federation & RM HA, the FEDERATION_FAILOVER_ENABLED indicates that we should configure failover RetryPolicies as otherwise normal ones (like retryForeverWithFixedSleep) would be initialized. This is needed because we override HA enabled in conf as otherwise that triggers the code path of looking up RMs from xml using rmId index.

          Why do we need to store the tokens and then re-add the tokens?

          This is to cover a corner case scenario which we hit during testing. IIRC, it was required for the UnmanagedAMs (used for transparently spanning a job across clusters) to reconnect to RM post failover. Without this, I was getting authentication error with missing token from the RM primary RM.
          Do you think I should leave it or add it when we reach the e2e testing phase?

          It appears to me the FederationStateStoreFacade is not needed for this patch

          Since we are using in-memory store for the test case, you are right. The issue is with the SQL based store (YARN-3663) which is what we want to deploy in production. We did some simulated scale testing with 10s of thousands of nodes and there was performance degradation without connection pooling. For pooling, you need Singleton as otherwise every proxy instance would create its own private pool and that's what FederationStateStoreFacade provides.

          Test: IIUC, I don't actually see FederationRMFailoverProxyProvider configured in the test, is it tested?

          Yes! In the test, I create a proxy using FederationProxyProviderUtil:

          ApplicationClientProtocol client = FederationProxyProviderUtil
          	        .createRMProxy(conf, ApplicationClientProtocol.class, subClusterId,
          	            UserGroupInformation.getCurrentUser());
          

          And FederationProxyProviderUtil always sets FederationRMFailoverProxyProvider as the failover provider:

          // updating the conf with the refreshed RM addresses as proxy creations
          	  // are based out of conf
          	  private static void updateConf(Configuration conf,
          	      SubClusterId subClusterId) {
          	    conf.set(YarnConfiguration.FEDERATION_SUBCLUSTER_ID, subClusterId.getId());
          	    conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true);
          	    conf.setClass(YarnConfiguration.CLIENT_FAILOVER_PROXY_PROVIDER,
          	        FederationRMFailoverProxyProvider.class, RMFailoverProxyProvider.class);
          	    // we will failover using FederationStateStore so set that and skip
          	    // traditional HA
          	    if (HAUtil.isHAEnabled(conf)) {
          	      conf.setBoolean(YarnConfiguration.FEDERATION_FAILOVER_ENABLED, true);
          	      conf.setBoolean(YarnConfiguration.RM_HA_ENABLED, false);
          	    }
          

          This also is how FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flags are used.

          And I had to do multiple rounds of debugging and step through FederationRMFailoverProxyProvider so many times while writing the tests, so most definitely it's being exercised .

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for reviewing the patch and your detailed feedback. I think for this patch the FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flag are not needed There is a nuance here that unlike the other failover providers which are only initialized when RM HA is enabled, FederationRMFailoverProxyProvider is used with both standalone RM and RM HA as the main purpose is to bypass yarn-site and obtain the RM connection address from FederationStateStore . The FEDERATION_ENABLED flag indicates that irrespective of whether RM has HA enabled or not, we should configure the FederationRMFailoverProxyProvider . When we have both Federation & RM HA, the FEDERATION_FAILOVER_ENABLED indicates that we should configure failover RetryPolicies as otherwise normal ones (like retryForeverWithFixedSleep ) would be initialized. This is needed because we override HA enabled in conf as otherwise that triggers the code path of looking up RMs from xml using rmId index. Why do we need to store the tokens and then re-add the tokens? This is to cover a corner case scenario which we hit during testing. IIRC, it was required for the UnmanagedAMs (used for transparently spanning a job across clusters) to reconnect to RM post failover. Without this, I was getting authentication error with missing token from the RM primary RM. Do you think I should leave it or add it when we reach the e2e testing phase? It appears to me the FederationStateStoreFacade is not needed for this patch Since we are using in-memory store for the test case, you are right. The issue is with the SQL based store ( YARN-3663 ) which is what we want to deploy in production. We did some simulated scale testing with 10s of thousands of nodes and there was performance degradation without connection pooling. For pooling, you need Singleton as otherwise every proxy instance would create its own private pool and that's what FederationStateStoreFacade provides. Test: IIUC, I don't actually see FederationRMFailoverProxyProvider configured in the test, is it tested? Yes! In the test, I create a proxy using FederationProxyProviderUtil : ApplicationClientProtocol client = FederationProxyProviderUtil .createRMProxy(conf, ApplicationClientProtocol.class, subClusterId, UserGroupInformation.getCurrentUser()); And FederationProxyProviderUtil always sets FederationRMFailoverProxyProvider as the failover provider: // updating the conf with the refreshed RM addresses as proxy creations // are based out of conf private static void updateConf(Configuration conf, SubClusterId subClusterId) { conf.set(YarnConfiguration.FEDERATION_SUBCLUSTER_ID, subClusterId.getId()); conf.setBoolean(YarnConfiguration.FEDERATION_ENABLED, true ); conf.setClass(YarnConfiguration.CLIENT_FAILOVER_PROXY_PROVIDER, FederationRMFailoverProxyProvider.class, RMFailoverProxyProvider.class); // we will failover using FederationStateStore so set that and skip // traditional HA if (HAUtil.isHAEnabled(conf)) { conf.setBoolean(YarnConfiguration.FEDERATION_FAILOVER_ENABLED, true ); conf.setBoolean(YarnConfiguration.RM_HA_ENABLED, false ); } This also is how FEDERATION_FAILOVER_ENABLED and FEDERATION_ENABLED flags are used. And I had to do multiple rounds of debugging and step through FederationRMFailoverProxyProvider so many times while writing the tests, so most definitely it's being exercised .
          Hide
          jianhe Jian He added a comment -
          • If FederationRMFailoverProxyProvider is used for standalone RM, we only have a single RM, how does it do failover ?
          • In the test, can you also validate that the RM address in the conf is updated properly when failover? also , add a logging in the performFailover method

            For pooling, you need Singleton

            My point is that it looks to me the cache in the singleton is not much used. Please correct me if I’m wrong. Because it always removes the object from the cache before doing failover.

          Show
          jianhe Jian He added a comment - If FederationRMFailoverProxyProvider is used for standalone RM, we only have a single RM, how does it do failover ? In the test, can you also validate that the RM address in the conf is updated properly when failover? also , add a logging in the performFailover method For pooling, you need Singleton My point is that it looks to me the cache in the singleton is not much used. Please correct me if I’m wrong. Because it always removes the object from the cache before doing failover.
          Hide
          subru Subru Krishnan added a comment -

          Updated patch (v2) that addresses Jian He's feedback and fix for Yetus warnings.

          If FederationRMFailoverProxyProvider is used for standalone RM, we only have a single RM, how does it do failover ?

          It does not do failover for standalone RM. It still obtains the address for the standalone RM from FederationStateStore and connects to it using a appropriate RetryPolicies (like retryForeverWithFixedSleep).

          My point is that it looks to me the cache in the singleton is not much used. Please correct me if I’m wrong. Because it always removes the object from the cache before doing failover.

          You are right, cache is not used for failover but pooling is independent of caching with Singleton access pattern the only commonality. Cache is across all store implementations (managed by facade) while pooling is dependent on store (SQL for example supports it). We are fine to bypass cache for failover as it's a rare event.

          Show
          subru Subru Krishnan added a comment - Updated patch (v2) that addresses Jian He 's feedback and fix for Yetus warnings. If FederationRMFailoverProxyProvider is used for standalone RM, we only have a single RM, how does it do failover ? It does not do failover for standalone RM. It still obtains the address for the standalone RM from FederationStateStore and connects to it using a appropriate RetryPolicies (like retryForeverWithFixedSleep). My point is that it looks to me the cache in the singleton is not much used. Please correct me if I’m wrong. Because it always removes the object from the cache before doing failover. You are right, cache is not used for failover but pooling is independent of caching with Singleton access pattern the only commonality. Cache is across all store implementations (managed by facade) while pooling is dependent on store (SQL for example supports it). We are fine to bypass cache for failover as it's a rare event.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 58s Maven dependency ordering for branch
          +1 mvninstall 8m 27s YARN-2915 passed
          +1 compile 2m 59s YARN-2915 passed
          +1 checkstyle 0m 45s YARN-2915 passed
          +1 mvnsite 2m 8s YARN-2915 passed
          +1 mvneclipse 0m 53s YARN-2915 passed
          +1 findbugs 3m 10s YARN-2915 passed
          +1 javadoc 1m 18s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 25s the patch passed
          +1 compile 2m 19s the patch passed
          +1 javac 2m 19s the patch passed
          -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235)
          +1 mvnsite 1m 37s the patch passed
          +1 mvneclipse 0m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 3s The patch has no ill-formed XML file.
          +1 findbugs 3m 28s the patch passed
          +1 javadoc 1m 8s the patch passed
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          +1 unit 0m 32s hadoop-yarn-server-common in the patch passed.
          -1 unit 15m 53s hadoop-yarn-client in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          52m 53s



          Reason Tests
          Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824623/YARN-3673-YARN-2915-v2.patch
          JIRA Issue YARN-3673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ccc7634e8858 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 8abb0a8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12835/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12835/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 58s Maven dependency ordering for branch +1 mvninstall 8m 27s YARN-2915 passed +1 compile 2m 59s YARN-2915 passed +1 checkstyle 0m 45s YARN-2915 passed +1 mvnsite 2m 8s YARN-2915 passed +1 mvneclipse 0m 53s YARN-2915 passed +1 findbugs 3m 10s YARN-2915 passed +1 javadoc 1m 18s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 25s the patch passed +1 compile 2m 19s the patch passed +1 javac 2m 19s the patch passed -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235) +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 3s The patch has no ill-formed XML file. +1 findbugs 3m 28s the patch passed +1 javadoc 1m 8s the patch passed +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 15s hadoop-yarn-common in the patch passed. +1 unit 0m 32s hadoop-yarn-server-common in the patch passed. -1 unit 15m 53s hadoop-yarn-client in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 52m 53s Reason Tests Failed junit tests hadoop.yarn.client.api.impl.TestYarnClient Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824623/YARN-3673-YARN-2915-v2.patch JIRA Issue YARN-3673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ccc7634e8858 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 8abb0a8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12835/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12835/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12835/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          The test failures are unrelated and are tracked in YARN-5389. I feel it's safe to ignore the remaining 2 checkstyle issues as it's caused by long config names.

          Show
          subru Subru Krishnan added a comment - The test failures are unrelated and are tracked in YARN-5389 . I feel it's safe to ignore the remaining 2 checkstyle issues as it's caused by long config names.
          Hide
          subru Subru Krishnan added a comment -

          Adding detailed code comments (v3) as suggested by Jian He

          Show
          subru Subru Krishnan added a comment - Adding detailed code comments (v3) as suggested by Jian He
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 59s Maven dependency ordering for branch
          +1 mvninstall 7m 35s YARN-2915 passed
          +1 compile 2m 52s YARN-2915 passed
          +1 checkstyle 0m 51s YARN-2915 passed
          +1 mvnsite 2m 5s YARN-2915 passed
          +1 mvneclipse 0m 58s YARN-2915 passed
          +1 findbugs 3m 45s YARN-2915 passed
          +1 javadoc 1m 29s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 54s the patch passed
          +1 compile 2m 45s the patch passed
          +1 javac 2m 45s the patch passed
          -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235)
          +1 mvnsite 1m 58s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 2s the patch passed
          +1 javadoc 1m 18s the patch passed
          +1 unit 0m 27s hadoop-yarn-api in the patch passed.
          +1 unit 2m 26s hadoop-yarn-common in the patch passed.
          +1 unit 0m 38s hadoop-yarn-server-common in the patch passed.
          +1 unit 16m 17s hadoop-yarn-client in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          55m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824749/YARN-3673-YARN-2915-v3.patch
          JIRA Issue YARN-3673
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 028a955fdd21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 8abb0a8
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12844/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12844/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12844/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 59s Maven dependency ordering for branch +1 mvninstall 7m 35s YARN-2915 passed +1 compile 2m 52s YARN-2915 passed +1 checkstyle 0m 51s YARN-2915 passed +1 mvnsite 2m 5s YARN-2915 passed +1 mvneclipse 0m 58s YARN-2915 passed +1 findbugs 3m 45s YARN-2915 passed +1 javadoc 1m 29s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 54s the patch passed +1 compile 2m 45s the patch passed +1 javac 2m 45s the patch passed -1 checkstyle 0m 42s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 235 unchanged - 0 fixed = 237 total (was 235) +1 mvnsite 1m 58s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 2s the patch passed +1 javadoc 1m 18s the patch passed +1 unit 0m 27s hadoop-yarn-api in the patch passed. +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 0m 38s hadoop-yarn-server-common in the patch passed. +1 unit 16m 17s hadoop-yarn-client in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 55m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824749/YARN-3673-YARN-2915-v3.patch JIRA Issue YARN-3673 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 028a955fdd21 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 8abb0a8 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12844/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12844/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12844/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Committed to YARN-2915 branch, thanks Subru !

          Show
          jianhe Jian He added a comment - Committed to YARN-2915 branch, thanks Subru !
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for the thoughtful review.

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for the thoughtful review.

            People

            • Assignee:
              subru Subru Krishnan
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development