Details

    • Hadoop Flags:
      Reviewed

      Description

      This JIRA proposes creating a facade for Federation State and Policy Store to simply access and have a common place for cache management etc that can be used by both Router & AMRMProxy

      1. YARN-3672-YARN-2915-v1.patch
        37 kB
        Subru Krishnan
      2. YARN-3672-YARN-2915-v2.patch
        43 kB
        Subru Krishnan
      3. YARN-3672-YARN-2915-v3.patch
        43 kB
        Subru Krishnan
      4. YARN-3672-YARN-2915-v4.patch
        41 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for reviewing/committing.

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for reviewing/committing.
          Hide
          jianhe Jian He added a comment -

          Committed to YARN-2915 branch, thanks Subru !

          Show
          jianhe Jian He added a comment - Committed to YARN-2915 branch, thanks Subru !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 43s Maven dependency ordering for branch
          +1 mvninstall 7m 39s YARN-2915 passed
          +1 compile 7m 11s YARN-2915 passed
          +1 checkstyle 1m 28s YARN-2915 passed
          +1 mvnsite 1m 34s YARN-2915 passed
          +1 mvneclipse 0m 49s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 2m 47s YARN-2915 passed
          +1 javadoc 1m 13s YARN-2915 passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 7m 11s the patch passed
          +1 javac 7m 11s the patch passed
          -1 checkstyle 1m 29s root: The patch generated 16 new + 214 unchanged - 0 fixed = 230 total (was 214)
          +1 mvnsite 1m 35s the patch passed
          +1 mvneclipse 0m 50s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 3m 8s the patch passed
          -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159)
          +1 unit 0m 12s hadoop-project in the patch passed.
          +1 unit 0m 28s hadoop-yarn-api in the patch passed.
          -1 unit 2m 19s hadoop-yarn-common in the patch failed.
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          46m 46s



          Reason Tests
          Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823778/YARN-3672-YARN-2915-v4.patch
          JIRA Issue YARN-3672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux c8e4f022302e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 0048ce1
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12779/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12779/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 43s Maven dependency ordering for branch +1 mvninstall 7m 39s YARN-2915 passed +1 compile 7m 11s YARN-2915 passed +1 checkstyle 1m 28s YARN-2915 passed +1 mvnsite 1m 34s YARN-2915 passed +1 mvneclipse 0m 49s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 2m 47s YARN-2915 passed +1 javadoc 1m 13s YARN-2915 passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 7m 11s the patch passed +1 javac 7m 11s the patch passed -1 checkstyle 1m 29s root: The patch generated 16 new + 214 unchanged - 0 fixed = 230 total (was 214) +1 mvnsite 1m 35s the patch passed +1 mvneclipse 0m 50s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 3m 8s the patch passed -1 javadoc 0m 18s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159) +1 unit 0m 12s hadoop-project in the patch passed. +1 unit 0m 28s hadoop-yarn-api in the patch passed. -1 unit 2m 19s hadoop-yarn-common in the patch failed. +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 46m 46s Reason Tests Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823778/YARN-3672-YARN-2915-v4.patch JIRA Issue YARN-3672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux c8e4f022302e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 0048ce1 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12779/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12779/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12779/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for your feedback.

          How will the RM failover play together with YARN-3673 ? Let’s say subCluster1(RM1, RM2), subCluster2(RM3, RM4). Looks like the implementation will ignore cluster intra-failover and do cluster inter-failover only ?

          The implementation will handle only cluster intra-failover as the RM failover proxy in YARN-3673 will be seeded based on subClusterId. The information on the StateStore will get updated as part of RM active services initialization (YARN-3671). In your example, the RM failover proxy will be a connection to subCluster1 which will initially point to say RM1 which is the current primary. Suppose RM1 fails over to RM2, RM2 will now heartbeat the StateStore against subCluster1 and we will auto-update the proxy to connect to RM2 (by querying getSubClusterInfo(subCluster1) on the Facade).
          The cluster inter-failover is determined by the policies (YARN-5323) as that defines how a queue spans multiple sub-clusters and the Router/AMRMProxy will create a RM failover proxy per subCluster in the policy.
          Makes sense?

          question for such API. It asks for a specific subCluster info, do we still need the filterInactiveSubClusters flag ? Even if it’s required, the behavior for the if/else is inconsistent, the if case is honoring the flag, while the else doesn’t.

          Good catch. I have removed filterInactiveSubClusters from getSubClusterInfo.

          I think we should not reuse these two configs for retry, the default value of both is zero.

          Valid point. I was trying to reuse existing configs as we have to add a few for Federation on top of the too many existing ones. I looked at the RMProxy and have replaced them with better fitting ones.

          I have updated the patch (v4) accordingly.

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for your feedback. How will the RM failover play together with YARN-3673 ? Let’s say subCluster1(RM1, RM2), subCluster2(RM3, RM4). Looks like the implementation will ignore cluster intra-failover and do cluster inter-failover only ? The implementation will handle only cluster intra-failover as the RM failover proxy in YARN-3673 will be seeded based on subClusterId . The information on the StateStore will get updated as part of RM active services initialization ( YARN-3671 ). In your example, the RM failover proxy will be a connection to subCluster1 which will initially point to say RM1 which is the current primary. Suppose RM1 fails over to RM2, RM2 will now heartbeat the StateStore against subCluster1 and we will auto-update the proxy to connect to RM2 (by querying getSubClusterInfo(subCluster1) on the Facade). The cluster inter-failover is determined by the policies ( YARN-5323 ) as that defines how a queue spans multiple sub-clusters and the Router/AMRMProxy will create a RM failover proxy per subCluster in the policy. Makes sense? question for such API. It asks for a specific subCluster info, do we still need the filterInactiveSubClusters flag ? Even if it’s required, the behavior for the if/else is inconsistent, the if case is honoring the flag, while the else doesn’t. Good catch. I have removed filterInactiveSubClusters from getSubClusterInfo. I think we should not reuse these two configs for retry, the default value of both is zero. Valid point. I was trying to reuse existing configs as we have to add a few for Federation on top of the too many existing ones. I looked at the RMProxy and have replaced them with better fitting ones. I have updated the patch (v4) accordingly.
          Hide
          jianhe Jian He added a comment -
          • How will the RM failover play together with YARN-3673 ? Let’s say subCluster1(RM1, RM2), subCluster2(RM3, RM4). Looks like the implementation will ignore cluster intra-failover and do cluster inter-failover only ?
          • question for such API. It asks for a specific subCluster info, do we still need the filterInactiveSubClusters flag ? Even if it’s required, the behavior for the if/else is inconsistent, the if case is honoring the flag, while the else doesn’t.
            public SubClusterInfo getSubCluster(final SubClusterId subClusterId,
                final boolean filterInactiveSubClusters) throws YarnException {
              if (isCachingEnabled()) {
                return getSubClusters(filterInactiveSubClusters).get(subClusterId);
              } else {
                return stateStore
                    .getSubCluster(GetSubClusterInfoRequest.newInstance(subClusterId))
                    .getSubClusterInfo();
              }
            }
            
          • I think we should not reuse these two configs for retry, the default value of both is zero.
            conf.getInt(YarnConfiguration.CLIENT_FAILOVER_RETRIES,
                YarnConfiguration.DEFAULT_CLIENT_FAILOVER_RETRIES),
            conf.getLong(
                YarnConfiguration.CLIENT_FAILOVER_RETRIES_ON_SOCKET_TIMEOUTS,
                YarnConfiguration.DEFAULT_CLIENT_FAILOVER_RETRIES_ON_SOCKET_TIMEOUTS),
            
          Show
          jianhe Jian He added a comment - How will the RM failover play together with YARN-3673 ? Let’s say subCluster1(RM1, RM2), subCluster2(RM3, RM4). Looks like the implementation will ignore cluster intra-failover and do cluster inter-failover only ? question for such API. It asks for a specific subCluster info, do we still need the filterInactiveSubClusters flag ? Even if it’s required, the behavior for the if/else is inconsistent, the if case is honoring the flag, while the else doesn’t. public SubClusterInfo getSubCluster( final SubClusterId subClusterId, final boolean filterInactiveSubClusters) throws YarnException { if (isCachingEnabled()) { return getSubClusters(filterInactiveSubClusters).get(subClusterId); } else { return stateStore .getSubCluster(GetSubClusterInfoRequest.newInstance(subClusterId)) .getSubClusterInfo(); } } I think we should not reuse these two configs for retry, the default value of both is zero. conf.getInt(YarnConfiguration.CLIENT_FAILOVER_RETRIES, YarnConfiguration.DEFAULT_CLIENT_FAILOVER_RETRIES), conf.getLong( YarnConfiguration.CLIENT_FAILOVER_RETRIES_ON_SOCKET_TIMEOUTS, YarnConfiguration.DEFAULT_CLIENT_FAILOVER_RETRIES_ON_SOCKET_TIMEOUTS),
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 19s Maven dependency ordering for branch
          +1 mvninstall 7m 17s YARN-2915 passed
          +1 compile 9m 4s YARN-2915 passed
          +1 checkstyle 1m 57s YARN-2915 passed
          +1 mvnsite 1m 58s YARN-2915 passed
          +1 mvneclipse 0m 56s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 3m 29s YARN-2915 passed
          +1 javadoc 1m 23s YARN-2915 passed
          0 mvndep 0m 18s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 7m 18s the patch passed
          +1 javac 7m 18s the patch passed
          -1 checkstyle 1m 35s root: The patch generated 16 new + 214 unchanged - 0 fixed = 230 total (was 214)
          +1 mvnsite 1m 42s the patch passed
          +1 mvneclipse 0m 55s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 3m 10s the patch passed
          -1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159)
          +1 unit 0m 9s hadoop-project in the patch passed.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed.
          -1 unit 2m 18s hadoop-yarn-common in the patch failed.
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          49m 18s



          Reason Tests
          Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823561/YARN-3672-YARN-2915-v3.patch
          JIRA Issue YARN-3672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux f4ebc472abd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / b689f55
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12764/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12764/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 7m 17s YARN-2915 passed +1 compile 9m 4s YARN-2915 passed +1 checkstyle 1m 57s YARN-2915 passed +1 mvnsite 1m 58s YARN-2915 passed +1 mvneclipse 0m 56s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 3m 29s YARN-2915 passed +1 javadoc 1m 23s YARN-2915 passed 0 mvndep 0m 18s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 7m 18s the patch passed +1 javac 7m 18s the patch passed -1 checkstyle 1m 35s root: The patch generated 16 new + 214 unchanged - 0 fixed = 230 total (was 214) +1 mvnsite 1m 42s the patch passed +1 mvneclipse 0m 55s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 3m 10s the patch passed -1 javadoc 0m 17s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159) +1 unit 0m 9s hadoop-project in the patch passed. +1 unit 0m 26s hadoop-yarn-api in the patch passed. -1 unit 2m 18s hadoop-yarn-common in the patch failed. +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 49m 18s Reason Tests Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823561/YARN-3672-YARN-2915-v3.patch JIRA Issue YARN-3672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux f4ebc472abd3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / b689f55 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12764/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12764/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12764/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Fixing the possible remaining Yetus warnings (v3).

          The test case failure is unrelated to the patch.

          Show
          subru Subru Krishnan added a comment - Fixing the possible remaining Yetus warnings (v3). The test case failure is unrelated to the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 21s Maven dependency ordering for branch
          +1 mvninstall 6m 48s YARN-2915 passed
          +1 compile 7m 45s YARN-2915 passed
          +1 checkstyle 1m 38s YARN-2915 passed
          +1 mvnsite 1m 41s YARN-2915 passed
          +1 mvneclipse 0m 51s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 3m 2s YARN-2915 passed
          +1 javadoc 1m 23s YARN-2915 passed
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 6m 49s the patch passed
          -1 javac 6m 49s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709)
          -1 checkstyle 1m 28s root: The patch generated 17 new + 214 unchanged - 0 fixed = 231 total (was 214)
          +1 mvnsite 1m 34s the patch passed
          +1 mvneclipse 0m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 4s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 3m 0s the patch passed
          -1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159)
          +1 unit 0m 10s hadoop-project in the patch passed.
          +1 unit 0m 25s hadoop-yarn-api in the patch passed.
          -1 unit 2m 17s hadoop-yarn-common in the patch failed.
          +1 unit 0m 32s hadoop-yarn-server-common in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          45m 7s



          Reason Tests
          Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823523/YARN-3672-YARN-2915-v2.patch
          JIRA Issue YARN-3672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 86707bf9dac4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / b689f55
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-checkstyle-root.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12759/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12759/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 6m 48s YARN-2915 passed +1 compile 7m 45s YARN-2915 passed +1 checkstyle 1m 38s YARN-2915 passed +1 mvnsite 1m 41s YARN-2915 passed +1 mvneclipse 0m 51s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 3m 2s YARN-2915 passed +1 javadoc 1m 23s YARN-2915 passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 6m 49s the patch passed -1 javac 6m 49s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709) -1 checkstyle 1m 28s root: The patch generated 17 new + 214 unchanged - 0 fixed = 231 total (was 214) +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 3m 0s the patch passed -1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159) +1 unit 0m 10s hadoop-project in the patch passed. +1 unit 0m 25s hadoop-yarn-api in the patch passed. -1 unit 2m 17s hadoop-yarn-common in the patch failed. +1 unit 0m 32s hadoop-yarn-server-common in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 45m 7s Reason Tests Failed junit tests hadoop.yarn.logaggregation.TestAggregatedLogFormat Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823523/YARN-3672-YARN-2915-v2.patch JIRA Issue YARN-3672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 86707bf9dac4 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / b689f55 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-checkstyle-root.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12759/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12759/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12759/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Attaching updated patch (v2) that fixes Yetus warnings and test case failures.

          I couldn't fix the javadoc/checkstyle warnings as I didn't find anything wrong.

          Show
          subru Subru Krishnan added a comment - Attaching updated patch (v2) that fixes Yetus warnings and test case failures. I couldn't fix the javadoc/checkstyle warnings as I didn't find anything wrong.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 45s Maven dependency ordering for branch
          +1 mvninstall 7m 19s YARN-2915 passed
          +1 compile 7m 21s YARN-2915 passed
          +1 checkstyle 1m 28s YARN-2915 passed
          +1 mvnsite 1m 2s YARN-2915 passed
          +1 mvneclipse 0m 36s YARN-2915 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 42s YARN-2915 passed
          +1 javadoc 0m 43s YARN-2915 passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 0m 49s the patch passed
          +1 compile 6m 46s the patch passed
          -1 javac 6m 46s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709)
          -1 checkstyle 1m 28s root: The patch generated 17 new + 208 unchanged - 0 fixed = 225 total (was 208)
          +1 mvnsite 1m 2s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project
          +1 findbugs 1m 56s the patch passed
          -1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159)
          +1 unit 0m 9s hadoop-project in the patch passed.
          -1 unit 0m 25s hadoop-yarn-api in the patch failed.
          -1 unit 0m 32s hadoop-yarn-server-common in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          38m 13s



          Reason Tests
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
            hadoop.yarn.server.federation.store.impl.TestMemoryFederationStateStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823163/YARN-3672-YARN-2915-v1.patch
          JIRA Issue YARN-3672
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 06d937a96bcb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / b689f55
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-checkstyle-root.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/whitespace-eol.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12739/testReport/
          modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: .
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12739/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 45s Maven dependency ordering for branch +1 mvninstall 7m 19s YARN-2915 passed +1 compile 7m 21s YARN-2915 passed +1 checkstyle 1m 28s YARN-2915 passed +1 mvnsite 1m 2s YARN-2915 passed +1 mvneclipse 0m 36s YARN-2915 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 42s YARN-2915 passed +1 javadoc 0m 43s YARN-2915 passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 0m 49s the patch passed +1 compile 6m 46s the patch passed -1 javac 6m 46s root generated 1 new + 709 unchanged - 0 fixed = 710 total (was 709) -1 checkstyle 1m 28s root: The patch generated 17 new + 208 unchanged - 0 fixed = 225 total (was 208) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 36s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-project +1 findbugs 1m 56s the patch passed -1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common generated 1 new + 159 unchanged - 0 fixed = 160 total (was 159) +1 unit 0m 9s hadoop-project in the patch passed. -1 unit 0m 25s hadoop-yarn-api in the patch failed. -1 unit 0m 32s hadoop-yarn-server-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 38m 13s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.federation.store.impl.TestMemoryFederationStateStore Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823163/YARN-3672-YARN-2915-v1.patch JIRA Issue YARN-3672 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 06d937a96bcb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / b689f55 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/whitespace-eol.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt https://builds.apache.org/job/PreCommit-YARN-Build/12739/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12739/testReport/ modules C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/12739/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Attaching a first version of FederationStateStoreFacade which is an utility wrapper that provides singleton access to the Federation state store. It abstracts out retries and in addition, it also implements the caching for various objects.

          This is also required by the proxy provider in YARN-3673 to connect to multiple RMs (subclusters).

          Show
          subru Subru Krishnan added a comment - Attaching a first version of FederationStateStoreFacade which is an utility wrapper that provides singleton access to the Federation state store. It abstracts out retries and in addition, it also implements the caching for various objects. This is also required by the proxy provider in YARN-3673 to connect to multiple RMs (subclusters).

            People

            • Assignee:
              subru Subru Krishnan
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development