Details

    • Hadoop Flags:
      Reviewed

      Description

      This JIRA proposes adding the ability to turn on Federation services like StateStore, cluster membership heartbeat etc in the RM

      1. YARN-3671-YARN-2915-v5.patch
        40 kB
        Subru Krishnan
      2. YARN-3671-YARN-2915-v4.patch
        40 kB
        Subru Krishnan
      3. YARN-3671-YARN-2915-v3.patch
        39 kB
        Subru Krishnan
      4. YARN-3671-YARN-2915-v2.patch
        37 kB
        Subru Krishnan
      5. YARN-3671-YARN-2915-v1.patch
        36 kB
        Subru Krishnan

        Issue Links

          Activity

          Hide
          subru Subru Krishnan added a comment -

          Attaching a patch that adds a new RMActiveService - FederationStateStoreService that's optionally turned on by a boolean FEDERATION_ENABLED flag. When enabled it starts a periodic heartbeat; FederationStateStoreHeartbeat to the FederationStateStore.

          Show
          subru Subru Krishnan added a comment - Attaching a patch that adds a new RMActiveService - FederationStateStoreService that's optionally turned on by a boolean FEDERATION_ENABLED flag. When enabled it starts a periodic heartbeat; FederationStateStoreHeartbeat to the FederationStateStore .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 0s Maven dependency ordering for branch
          +1 mvninstall 6m 48s YARN-2915 passed
          +1 compile 2m 19s YARN-2915 passed
          +1 checkstyle 0m 42s YARN-2915 passed
          +1 mvnsite 1m 28s YARN-2915 passed
          +1 mvneclipse 0m 41s YARN-2915 passed
          +1 findbugs 2m 37s YARN-2915 passed
          +1 javadoc 0m 54s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 12s the patch passed
          +1 compile 2m 15s the patch passed
          +1 javac 2m 15s the patch passed
          -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 288 unchanged - 1 fixed = 291 total (was 289)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 56s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          +1 unit 0m 32s hadoop-yarn-server-common in the patch passed.
          +1 unit 37m 27s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          66m 23s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825381/YARN-3671-YARN-2915-v1.patch
          JIRA Issue YARN-3671
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b973b3621daa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 256034d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12886/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12886/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12886/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 0s Maven dependency ordering for branch +1 mvninstall 6m 48s YARN-2915 passed +1 compile 2m 19s YARN-2915 passed +1 checkstyle 0m 42s YARN-2915 passed +1 mvnsite 1m 28s YARN-2915 passed +1 mvneclipse 0m 41s YARN-2915 passed +1 findbugs 2m 37s YARN-2915 passed +1 javadoc 0m 54s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 2m 15s the patch passed +1 javac 2m 15s the patch passed -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 288 unchanged - 1 fixed = 291 total (was 289) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 56s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 0m 22s hadoop-yarn-api in the patch passed. +1 unit 0m 32s hadoop-yarn-server-common in the patch passed. +1 unit 37m 27s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 66m 23s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825381/YARN-3671-YARN-2915-v1.patch JIRA Issue YARN-3671 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b973b3621daa 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 256034d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12886/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12886/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12886/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -
          • we already have RM_CLUSTER_ID, any chance that this can be used for FEDERATION_SUBCLUSTER_ID ?
          • This null check is unnecessary.
            federationStateStoreService = createFederationStateStoreService();
            
if (federationStateStoreService != null) {
            
          • I feel the SubClusterState is a bit redundant in the request object, because the API itself already indicates the state such as register / deregister
          • why you need to call reinitialize ?
            public void setStateStoreClient(FederationStateStore stateStore) {
  
            this.stateStoreClient = stateStore;
 
             reInitialize();
            
}
            
          Show
          jianhe Jian He added a comment - we already have RM_CLUSTER_ID, any chance that this can be used for FEDERATION_SUBCLUSTER_ID ? This null check is unnecessary. federationStateStoreService = createFederationStateStoreService(); 
 if (federationStateStoreService != null ) { I feel the SubClusterState is a bit redundant in the request object, because the API itself already indicates the state such as register / deregister why you need to call reinitialize ? public void setStateStoreClient(FederationStateStore stateStore) {
 this .stateStoreClient = stateStore;
 reInitialize(); 
}
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for the feedback. Updated patch (v2) to remove redundant null check and refactor setStateStoreClient as suggested by you.

          As to your other questions:

          we already have RM_CLUSTER_ID, any chance that this can be used for FEDERATION_SUBCLUSTER_ID ?

          That's a possibility. The reason I didn't combine both is RM_CLUSTER_ID is currently used for HA but Federation can work both with and without HA (and RM HA can work both with and without Federation). So felt it would be better to keep them separate. Thoughts?

          I feel the SubClusterState is a bit redundant in the request object, because the API itself already indicates the state such as register / deregister.

          You are right. We don't want state to be null in the store so either the store impl can implicitly add SC_NEW/SC_UNREGISTERED on register / deregister or the invoker (which is always RM) can. I decided to do it in the RM for 2 reasons:
          1. It is trivial (one line) & needs to be done in a single place (RM) instead of in each store impl we add.
          2. This allows for flexibility future as RM could potentially register / deregister with different states (say SC_DRAINING).

          Makes sense?

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for the feedback. Updated patch (v2) to remove redundant null check and refactor setStateStoreClient as suggested by you. As to your other questions: we already have RM_CLUSTER_ID, any chance that this can be used for FEDERATION_SUBCLUSTER_ID ? That's a possibility. The reason I didn't combine both is RM_CLUSTER_ID is currently used for HA but Federation can work both with and without HA (and RM HA can work both with and without Federation). So felt it would be better to keep them separate. Thoughts? I feel the SubClusterState is a bit redundant in the request object, because the API itself already indicates the state such as register / deregister. You are right. We don't want state to be null in the store so either the store impl can implicitly add SC_NEW/SC_UNREGISTERED on register / deregister or the invoker (which is always RM) can. I decided to do it in the RM for 2 reasons: 1. It is trivial (one line) & needs to be done in a single place (RM) instead of in each store impl we add. 2. This allows for flexibility future as RM could potentially register / deregister with different states (say SC_DRAINING). Makes sense?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 9s Maven dependency ordering for branch
          +1 mvninstall 6m 42s YARN-2915 passed
          +1 compile 2m 19s YARN-2915 passed
          +1 checkstyle 0m 41s YARN-2915 passed
          +1 mvnsite 1m 28s YARN-2915 passed
          +1 mvneclipse 0m 41s YARN-2915 passed
          +1 findbugs 2m 37s YARN-2915 passed
          +1 javadoc 0m 54s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 11s the patch passed
          +1 compile 2m 16s the patch passed
          +1 javac 2m 16s the patch passed
          -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 288 unchanged - 1 fixed = 291 total (was 289)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 57s the patch passed
          +1 javadoc 0m 48s the patch passed
          +1 unit 0m 22s hadoop-yarn-api in the patch passed.
          +1 unit 0m 31s hadoop-yarn-server-common in the patch passed.
          +1 unit 38m 29s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          66m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825543/YARN-3671-YARN-2915-v2.patch
          JIRA Issue YARN-3671
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 956ad14fb536 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 256034d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12901/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12901/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12901/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 42s YARN-2915 passed +1 compile 2m 19s YARN-2915 passed +1 checkstyle 0m 41s YARN-2915 passed +1 mvnsite 1m 28s YARN-2915 passed +1 mvneclipse 0m 41s YARN-2915 passed +1 findbugs 2m 37s YARN-2915 passed +1 javadoc 0m 54s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 2m 16s the patch passed +1 javac 2m 16s the patch passed -1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 288 unchanged - 1 fixed = 291 total (was 289) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 57s the patch passed +1 javadoc 0m 48s the patch passed +1 unit 0m 22s hadoop-yarn-api in the patch passed. +1 unit 0m 31s hadoop-yarn-server-common in the patch passed. +1 unit 38m 29s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 66m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825543/YARN-3671-YARN-2915-v2.patch JIRA Issue YARN-3671 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 956ad14fb536 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 256034d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12901/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12901/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12901/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          RM_CLUSTER_ID is currently used for HA but Federation can work both with and without HA

          RM_CLUSTER_ID is not used in non-HA, because it's not needed. It can also be used without HA. My take is that it seems two configures for the same purpose to identify a cluster, or I missed certain use-case ?

          Show
          jianhe Jian He added a comment - RM_CLUSTER_ID is currently used for HA but Federation can work both with and without HA RM_CLUSTER_ID is not used in non-HA, because it's not needed. It can also be used without HA. My take is that it seems two configures for the same purpose to identify a cluster, or I missed certain use-case ?
          Hide
          subru Subru Krishnan added a comment -

          Jian He, I looked into it & seems like we should be able to reuse RM_CLUSTER_ID safely as you suggested. So I have replaced FEDERATION_SUBCLUSTER_ID with RM_CLUSTER_ID in v3 of the patch.

          Show
          subru Subru Krishnan added a comment - Jian He , I looked into it & seems like we should be able to reuse RM_CLUSTER_ID safely as you suggested. So I have replaced FEDERATION_SUBCLUSTER_ID with RM_CLUSTER_ID in v3 of the patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 55s Maven dependency ordering for branch
          +1 mvninstall 6m 51s YARN-2915 passed
          +1 compile 2m 24s YARN-2915 passed
          +1 checkstyle 0m 47s YARN-2915 passed
          +1 mvnsite 1m 41s YARN-2915 passed
          +1 mvneclipse 0m 45s YARN-2915 passed
          +1 findbugs 3m 8s YARN-2915 passed
          +1 javadoc 1m 0s YARN-2915 passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 1m 26s the patch passed
          +1 compile 2m 50s the patch passed
          +1 javac 2m 50s the patch passed
          -1 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 290 unchanged - 1 fixed = 293 total (was 291)
          +1 mvnsite 1m 36s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 0s the patch passed
          +1 javadoc 0m 49s the patch passed
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 0m 33s hadoop-yarn-server-common in the patch passed.
          +1 unit 37m 24s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          68m 42s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825702/YARN-3671-YARN-2915-v3.patch
          JIRA Issue YARN-3671
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5b0280fb78b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 256034d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12905/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12905/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12905/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 55s Maven dependency ordering for branch +1 mvninstall 6m 51s YARN-2915 passed +1 compile 2m 24s YARN-2915 passed +1 checkstyle 0m 47s YARN-2915 passed +1 mvnsite 1m 41s YARN-2915 passed +1 mvneclipse 0m 45s YARN-2915 passed +1 findbugs 3m 8s YARN-2915 passed +1 javadoc 1m 0s YARN-2915 passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 2m 50s the patch passed +1 javac 2m 50s the patch passed -1 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 290 unchanged - 1 fixed = 293 total (was 291) +1 mvnsite 1m 36s the patch passed +1 mvneclipse 0m 37s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 0s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 0m 33s hadoop-yarn-server-common in the patch passed. +1 unit 37m 24s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 68m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825702/YARN-3671-YARN-2915-v3.patch JIRA Issue YARN-3671 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5b0280fb78b8 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 256034d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12905/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12905/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12905/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Thanks Subru for updating !

          • Here, the scheduler is passed in via reference. If the RM switches to standby and then switches back to active, the scheduler object will be re-created and the reference will be updated in RMContext, but the reference here won't be updated, which cause the scheduler object here outdated?
              public FederationStateStoreHeartbeat(FederationStateStore stateStoreClient,
                  ResourceScheduler scheduler, SubClusterId clusterId) {
                this.stateStoreService = stateStoreClient;
                this.subClusterId = clusterId;
                this.rs = scheduler;
            
          • "state-store.heartbeat-interval": how about "state-store.heartbeat-interval-secs"
          Show
          jianhe Jian He added a comment - Thanks Subru for updating ! Here, the scheduler is passed in via reference. If the RM switches to standby and then switches back to active, the scheduler object will be re-created and the reference will be updated in RMContext, but the reference here won't be updated, which cause the scheduler object here outdated? public FederationStateStoreHeartbeat(FederationStateStore stateStoreClient, ResourceScheduler scheduler, SubClusterId clusterId) { this .stateStoreService = stateStoreClient; this .subClusterId = clusterId; this .rs = scheduler; "state-store.heartbeat-interval": how about "state-store.heartbeat-interval-secs"
          Hide
          subru Subru Krishnan added a comment -

          Here, the scheduler is passed in via reference. If the RM switches to standby and then switches back to active, the scheduler object will be re-created and the reference will be updated in RMContext, but the reference here won't be updated, which cause the scheduler object here outdated?

          Good question Jian He. The scheduler object will not be outdated as FederationStateStoreService is an active service and the FederationStateStoreHeartbeat is initialized on it's serviceStart:

          stateStoreHeartbeat = new FederationStateStoreHeartbeat(subClusterId,
                  stateStoreClient, rmContext.getScheduler());
          

          To double-check, I have updated the TestFederationRMStateStoreService with an explicit transtion to standby and then back to active.

          I have also renamed "state-store.heartbeat-interval" --> "state-store.heartbeat-interval-secs" in v4 of the patch as you suggested.

          Show
          subru Subru Krishnan added a comment - Here, the scheduler is passed in via reference. If the RM switches to standby and then switches back to active, the scheduler object will be re-created and the reference will be updated in RMContext, but the reference here won't be updated, which cause the scheduler object here outdated? Good question Jian He . The scheduler object will not be outdated as FederationStateStoreService is an active service and the FederationStateStoreHeartbeat is initialized on it's serviceStart : stateStoreHeartbeat = new FederationStateStoreHeartbeat(subClusterId, stateStoreClient, rmContext.getScheduler()); To double-check, I have updated the TestFederationRMStateStoreService with an explicit transtion to standby and then back to active. I have also renamed "state-store.heartbeat-interval" --> "state-store.heartbeat-interval-secs" in v4 of the patch as you suggested.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 0s Maven dependency ordering for branch
          +1 mvninstall 7m 44s YARN-2915 passed
          +1 compile 2m 20s YARN-2915 passed
          +1 checkstyle 0m 42s YARN-2915 passed
          +1 mvnsite 1m 28s YARN-2915 passed
          +1 mvneclipse 0m 42s YARN-2915 passed
          +1 findbugs 2m 38s YARN-2915 passed
          +1 javadoc 0m 56s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 2m 39s the patch passed
          +1 javac 2m 39s the patch passed
          -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 290 unchanged - 1 fixed = 296 total (was 291)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 51s the patch passed
          +1 javadoc 1m 1s the patch passed
          +1 unit 0m 31s hadoop-yarn-api in the patch passed.
          +1 unit 0m 40s hadoop-yarn-server-common in the patch passed.
          +1 unit 34m 51s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          66m 54s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826081/YARN-3671-YARN-2915-v4.patch
          JIRA Issue YARN-3671
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 5a2a16803114 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 256034d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12931/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12931/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12931/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 0s Maven dependency ordering for branch +1 mvninstall 7m 44s YARN-2915 passed +1 compile 2m 20s YARN-2915 passed +1 checkstyle 0m 42s YARN-2915 passed +1 mvnsite 1m 28s YARN-2915 passed +1 mvneclipse 0m 42s YARN-2915 passed +1 findbugs 2m 38s YARN-2915 passed +1 javadoc 0m 56s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 2m 39s the patch passed +1 javac 2m 39s the patch passed -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 290 unchanged - 1 fixed = 296 total (was 291) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 51s the patch passed +1 javadoc 1m 1s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 0m 40s hadoop-yarn-server-common in the patch passed. +1 unit 34m 51s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 66m 54s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826081/YARN-3671-YARN-2915-v4.patch JIRA Issue YARN-3671 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 5a2a16803114 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 256034d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12931/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12931/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12931/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          subru Subru Krishnan added a comment -

          Fixing the checkstyle warning that can be addressed (v5).

          Show
          subru Subru Krishnan added a comment - Fixing the checkstyle warning that can be addressed (v5).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 docker 6m 37s Docker failed to build yetus/hadoop:9560f25.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826095/YARN-3671-YARN-2915-v5.patch
          JIRA Issue YARN-3671
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12935/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 6m 37s Docker failed to build yetus/hadoop:9560f25. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826095/YARN-3671-YARN-2915-v5.patch JIRA Issue YARN-3671 Console output https://builds.apache.org/job/PreCommit-YARN-Build/12935/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 59s Maven dependency ordering for branch
          +1 mvninstall 6m 55s YARN-2915 passed
          +1 compile 2m 19s YARN-2915 passed
          +1 checkstyle 0m 43s YARN-2915 passed
          +1 mvnsite 1m 28s YARN-2915 passed
          +1 mvneclipse 0m 42s YARN-2915 passed
          +1 findbugs 2m 42s YARN-2915 passed
          +1 javadoc 0m 55s YARN-2915 passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 15s the patch passed
          +1 compile 2m 29s the patch passed
          +1 javac 2m 29s the patch passed
          -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 290 unchanged - 1 fixed = 295 total (was 291)
          +1 mvnsite 1m 25s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 10s the patch passed
          +1 javadoc 0m 50s the patch passed
          +1 unit 0m 25s hadoop-yarn-api in the patch passed.
          +1 unit 0m 34s hadoop-yarn-server-common in the patch passed.
          +1 unit 33m 48s hadoop-yarn-server-resourcemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          63m 38s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826095/YARN-3671-YARN-2915-v5.patch
          JIRA Issue YARN-3671
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4ff776e770fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision YARN-2915 / 256034d
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12937/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12937/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12937/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 59s Maven dependency ordering for branch +1 mvninstall 6m 55s YARN-2915 passed +1 compile 2m 19s YARN-2915 passed +1 checkstyle 0m 43s YARN-2915 passed +1 mvnsite 1m 28s YARN-2915 passed +1 mvneclipse 0m 42s YARN-2915 passed +1 findbugs 2m 42s YARN-2915 passed +1 javadoc 0m 55s YARN-2915 passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 2m 29s the patch passed +1 javac 2m 29s the patch passed -1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 290 unchanged - 1 fixed = 295 total (was 291) +1 mvnsite 1m 25s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 10s the patch passed +1 javadoc 0m 50s the patch passed +1 unit 0m 25s hadoop-yarn-api in the patch passed. +1 unit 0m 34s hadoop-yarn-server-common in the patch passed. +1 unit 33m 48s hadoop-yarn-server-resourcemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 63m 38s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826095/YARN-3671-YARN-2915-v5.patch JIRA Issue YARN-3671 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4ff776e770fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision YARN-2915 / 256034d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/12937/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12937/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12937/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jianhe Jian He added a comment -

          Looks good,

          Committed to YARN-2915. Thanks Subru !

          Show
          jianhe Jian He added a comment - Looks good, Committed to YARN-2915 . Thanks Subru !
          Hide
          subru Subru Krishnan added a comment -

          Thanks Jian He for the thoughtful review.

          Show
          subru Subru Krishnan added a comment - Thanks Jian He for the thoughtful review.

            People

            • Assignee:
              subru Subru Krishnan
              Reporter:
              subru Subru Krishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development