Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3623

We should have a config to indicate the Timeline Service version

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: timelineserver
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Add a new configuration "yarn.timeline-service.version" to indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and should bring up the timeline service v.1.5. On the client side, if the client uses the same version of timeline service, it should succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary.
      Show
      Add a new configuration "yarn.timeline-service.version" to indicate what is the current version of the running timeline service. For example, if "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it means the cluster will and should bring up the timeline service v.1.5. On the client side, if the client uses the same version of timeline service, it should succeed. If the client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary.

      Description

      So far RM, MR AM, DA AM added/changed new config to enable the feature to write the timeline data to v2 server. It's good to have a YARN timeline-service.version config like timeline-service.enable to indicate the version of the running timeline service with the given YARN cluster. It's beneficial for users to more smoothly move from v1 to v2, as they don't need to change the existing config, but switch this config from v1 to v2. And each framework doesn't need to have their own v1/v2 config.

      1. YARN-3623-2015-11-19.1.patch
        2 kB
        Xuan Gong
      2. YARN-3623-2015-12-09.patch
        2 kB
        Xuan Gong
      3. YARN-3623-addendum.patch
        1.0 kB
        Junping Du

        Issue Links

          Activity

          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Zhijie Shen & Sangjin Lee, Further to the above mentioned scenario, i think it was decided we are going to support both V1 along with V2. Earlier IIRC we had not gone for V1 and V2 configuration because if in future V1 is removed then this configuration will become redundant but based on later discussions we decide to have both right ? so shall we get these changes done ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Zhijie Shen & Sangjin Lee , Further to the above mentioned scenario, i think it was decided we are going to support both V1 along with V2. Earlier IIRC we had not gone for V1 and V2 configuration because if in future V1 is removed then this configuration will become redundant but based on later discussions we decide to have both right ? so shall we get these changes done ?
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          I would like to handle this jira once Version config is pushed in as part of 1.5.

          Show
          Naganarasimha Naganarasimha G R added a comment - I would like to handle this jira once Version config is pushed in as part of 1.5.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Xuan Gong / Naganarasimha G R / Sangjin Lee / Li Lu, there are too many JIRAs about this versioning topic and the discussion is split all over.

          Let me try to consolidate the tickets. Let's use this one as the base JIRA for introducing versioning. Xuan Gong, can you post part of your patch at YARN-4234 here?

          I'll move this to be a common JIRA across v1, v2 and v1.5 and make all the other JIRAs depend on this.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Xuan Gong / Naganarasimha G R / Sangjin Lee / Li Lu , there are too many JIRAs about this versioning topic and the discussion is split all over. Let me try to consolidate the tickets. Let's use this one as the base JIRA for introducing versioning. Xuan Gong , can you post part of your patch at YARN-4234 here? I'll move this to be a common JIRA across v1, v2 and v1.5 and make all the other JIRAs depend on this.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Copying comments from YARN-4183 w.r.t versioning:

          By Me

          • There should be an explicit yarn.timeline-service.version which tells YarnClient to get tokens or not - yes for non-present version (default), v1, v2 but no for v1.5.
          • We should also use the same property in the new API calls proposed for V1.5 YARN-4233 / V2 YARN-2928, lest the users think they can call any API independent of what is supported on server side.
          • The version field has semantics on both client and server side at the same time - it's picking a solution end-to-end.

          By Sangjin Lee

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Copying comments from YARN-4183 w.r.t versioning: By Me There should be an explicit yarn.timeline-service.version which tells YarnClient to get tokens or not - yes for non-present version (default), v1, v2 but no for v1.5. We should also use the same property in the new API calls proposed for V1.5 YARN-4233 / V2 YARN-2928 , lest the users think they can call any API independent of what is supported on server side. The version field has semantics on both client and server side at the same time - it's picking a solution end-to-end. By Sangjin Lee The version is a list or enum - https://issues.apache.org/jira/browse/YARN-4183?focusedCommentId=15004954&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15004954 . Related JIRA: YARN-4368 .
          Hide
          xgong Xuan Gong added a comment -

          Attached a simple patch which added a new config to indicate the Timeline Service version.

          Show
          xgong Xuan Gong added a comment - Attached a simple patch which added a new config to indicate the Timeline Service version.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 10m 44s trunk passed
          +1 compile 2m 42s trunk passed with JDK v1.8.0_66
          +1 compile 2m 34s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          -1 findbugs 1m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings.
          +1 javadoc 1m 18s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 37s trunk passed with JDK v1.7.0_85
          +1 mvninstall 1m 7s the patch passed
          +1 compile 2m 20s the patch passed with JDK v1.8.0_66
          +1 javac 2m 20s the patch passed
          +1 compile 2m 29s the patch passed with JDK v1.7.0_85
          +1 javac 2m 29s the patch passed
          -1 checkstyle 0m 34s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 212, now 212).
          +1 mvnsite 1m 12s the patch passed
          +1 mvneclipse 0m 28s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 33s the patch passed
          +1 javadoc 1m 26s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 49s the patch passed with JDK v1.7.0_85
          +1 unit 0m 31s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          -1 unit 2m 21s hadoop-yarn-common in the patch failed with JDK v1.8.0_66.
          +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_85.
          -1 unit 2m 26s hadoop-yarn-common in the patch failed with JDK v1.7.0_85.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          50m 52s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.yarn.webapp.TestWebApp
          JDK v1.7.0_85 Failed junit tests hadoop.yarn.webapp.TestWebApp



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:date2015-11-20
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773373/YARN-3623-2015-11-19.1.patch
          JIRA Issue YARN-3623
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux e4b88f440a7c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh
          git revision trunk / 4539131
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9745/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn
          Max memory used 78MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9745/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 10m 44s trunk passed +1 compile 2m 42s trunk passed with JDK v1.8.0_66 +1 compile 2m 34s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 27s trunk passed -1 findbugs 1m 30s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common in trunk has 3 extant Findbugs warnings. +1 javadoc 1m 18s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 37s trunk passed with JDK v1.7.0_85 +1 mvninstall 1m 7s the patch passed +1 compile 2m 20s the patch passed with JDK v1.8.0_66 +1 javac 2m 20s the patch passed +1 compile 2m 29s the patch passed with JDK v1.7.0_85 +1 javac 2m 29s the patch passed -1 checkstyle 0m 34s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 212, now 212). +1 mvnsite 1m 12s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 33s the patch passed +1 javadoc 1m 26s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 49s the patch passed with JDK v1.7.0_85 +1 unit 0m 31s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. -1 unit 2m 21s hadoop-yarn-common in the patch failed with JDK v1.8.0_66. +1 unit 0m 29s hadoop-yarn-api in the patch passed with JDK v1.7.0_85. -1 unit 2m 26s hadoop-yarn-common in the patch failed with JDK v1.7.0_85. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 50m 52s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.yarn.webapp.TestWebApp JDK v1.7.0_85 Failed junit tests hadoop.yarn.webapp.TestWebApp Subsystem Report/Notes Docker Image:yetus/hadoop:date2015-11-20 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773373/YARN-3623-2015-11-19.1.patch JIRA Issue YARN-3623 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e4b88f440a7c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-YARN-Build/patchprocess/apache-yetus-3f4279a/precommit/personality/hadoop.sh git revision trunk / 4539131 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-YARN-Build/9745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9745/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn Max memory used 78MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9745/console This message was automatically generated.
          Hide
          gtCarrera9 Li Lu added a comment -

          Patch LGTM. The findbugs warnings appears to be irrelevant to the fix here.

          Show
          gtCarrera9 Li Lu added a comment - Patch LGTM. The findbugs warnings appears to be irrelevant to the fix here.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          This would be good Vinod Kumar Vavilapalli,
          but was under the impression that timeline service version is not required for versions before ATS 1.5 and this would be introduced in YARN-4234 (as part of ATS 1.5) and the required modifications where ever applicable in ATSv2 would be handled in this jira (earlier ATS subjira). Ok anyway can create another subjira in 2928 to handle usage of this newly introduced version.

          Show
          Naganarasimha Naganarasimha G R added a comment - This would be good Vinod Kumar Vavilapalli , but was under the impression that timeline service version is not required for versions before ATS 1.5 and this would be introduced in YARN-4234 (as part of ATS 1.5) and the required modifications where ever applicable in ATSv2 would be handled in this jira (earlier ATS subjira). Ok anyway can create another subjira in 2928 to handle usage of this newly introduced version.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Vinod Kumar Vavilapalli,

          There should be an explicit yarn.timeline-service.version which tells YarnClient to get tokens or not - yes for non-present version (default), v1, v2 but no for v1.5.

          IIUC this contradicts the Sangjin Lee's proposal which was discussed in YARN-4183 and the approach which i had discussed in YARN-4234 with Li Lu (inactive),
          I can envisage it as follows (including the opinions of Sangjin Lee):

          • timeline service will be configured with "yarn.timeline-service.enabled" and "TIMELINE_SERVICE_VERSION" to be true based on which appropriate timeline handler is selected
          • RM/NM (SMP) will be configured with "yarn.timeline-service.enabled" and yarn.resourcemanager.system-metrics-publisher.enabled to be set to true and along with it "TIMELINE_SERVICE_VERSION" based on which appropriate client will be selected.
          • Client apps who ever want to communicate with Timeline server will enable "yarn.timeline-service.enabled" and a new config like yarn.timeline-service.client.require-delegation-token to be set to true to publish the ATS events.
          • If security , "yarn.timeline-service.enabled" and "yarn.timeline-service.client.require-delegation-token" are enabled then delegation token is automatically fetched in the yarn client as earlier
          • when the timeline client is initialized it contacts server for the version and related configs through REST and once it receives it initializes itself. If user tries to use invalid methods (not appropriate to the server timeline version) then timelineclient throws exception.

          Thus this will avoid improper configurations between client and server and also allows client to have flexible configurations from server to avoid getting delegation tokens from ATS based on server config.
          A note : In the above proposal have not considered the multiple versions which is still under discussion in YARN-4368

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Vinod Kumar Vavilapalli , There should be an explicit yarn.timeline-service.version which tells YarnClient to get tokens or not - yes for non-present version (default), v1, v2 but no for v1.5. IIUC this contradicts the Sangjin Lee 's proposal which was discussed in YARN-4183 and the approach which i had discussed in YARN-4234 with Li Lu (inactive) , I can envisage it as follows (including the opinions of Sangjin Lee ): timeline service will be configured with "yarn.timeline-service.enabled" and "TIMELINE_SERVICE_VERSION" to be true based on which appropriate timeline handler is selected RM/NM (SMP) will be configured with "yarn.timeline-service.enabled" and yarn.resourcemanager.system-metrics-publisher.enabled to be set to true and along with it "TIMELINE_SERVICE_VERSION" based on which appropriate client will be selected. Client apps who ever want to communicate with Timeline server will enable "yarn.timeline-service.enabled" and a new config like yarn.timeline-service.client.require-delegation-token to be set to true to publish the ATS events. If security , "yarn.timeline-service.enabled" and "yarn.timeline-service.client.require-delegation-token" are enabled then delegation token is automatically fetched in the yarn client as earlier when the timeline client is initialized it contacts server for the version and related configs through REST and once it receives it initializes itself. If user tries to use invalid methods (not appropriate to the server timeline version) then timelineclient throws exception. Thus this will avoid improper configurations between client and server and also allows client to have flexible configurations from server to avoid getting delegation tokens from ATS based on server config. A note : In the above proposal have not considered the multiple versions which is still under discussion in YARN-4368
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Vinod Kumar Vavilapalli, Sangjin Lee, Li Lu (inactive) & Xuan Gong,
          If the above solution is fine then we can have following steps

          1. make this jira as subjira of 1.5 and introduce the timeline version
          2. As part of YARN-4183, we can create "yarn.timeline-service.client.require-delegation-token" so that it fixes depency on "yarn.timeline-service.enabled" in the client side to get tokens
          3. YARN-4356 or a new jira can handle modifications and updates of timeline version in ATSv2
          4. Can raise new jira and handle REST interface support to get the supported ATS version and config from the server for ATS 1.5
          5. Can raise new jira and handle version checking for the ATSv2 interface methods in TimelineClient.and also fetching of version
          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Vinod Kumar Vavilapalli , Sangjin Lee , Li Lu (inactive) & Xuan Gong , If the above solution is fine then we can have following steps make this jira as subjira of 1.5 and introduce the timeline version As part of YARN-4183 , we can create "yarn.timeline-service.client.require-delegation-token" so that it fixes depency on "yarn.timeline-service.enabled" in the client side to get tokens YARN-4356 or a new jira can handle modifications and updates of timeline version in ATSv2 Can raise new jira and handle REST interface support to get the supported ATS version and config from the server for ATS 1.5 Can raise new jira and handle version checking for the ATSv2 interface methods in TimelineClient.and also fetching of version
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi Naganarasimha G R sorry for the late reply (just came back from a vacation). Plan sounds good to me. One thing is that we may not need to mark this JIRA as ATS v1.5 since the fix here is a rather general one: from v1.5 on we need to handle this configuration in ATS correctly, thus I think it'll be a general JIRA and not attached with any specific version of ATS. Right now we have the patch for this JIRA so we can proceed with the rest of the plan? It will certainly be helpful if anyone has any concerns on Naga's plan.

          Show
          gtCarrera9 Li Lu added a comment - Hi Naganarasimha G R sorry for the late reply (just came back from a vacation). Plan sounds good to me. One thing is that we may not need to mark this JIRA as ATS v1.5 since the fix here is a rather general one: from v1.5 on we need to handle this configuration in ATS correctly, thus I think it'll be a general JIRA and not attached with any specific version of ATS. Right now we have the patch for this JIRA so we can proceed with the rest of the plan? It will certainly be helpful if anyone has any concerns on Naga's plan.
          Hide
          sjlee0 Sangjin Lee added a comment -

          4. Can raise new jira and handle REST interface support to get the supported ATS version and config from the server for ATS 1.5
          5. Can raise new jira and handle version checking for the ATSv2 interface methods in TimelineClient.and also fetching of version

          In my mind, trying to get the version by querying the REST interface seems highly problematic. If we're going to have a configuration that clearly spells out which version the cluster is running, why not rely on that config? Is there a reason to believe that may not be correct?

          In terms of issues of querying the REST interface for the version, first, the write URLs are not really feasible. V.1.5 may not even have the write REST endpoint up. For v.2, you really need to first discover the right URL (i.e. server) to talk to. You do it anyway, but that's only to know which endpoint to talk to, not to discover which version it needs to use.

          Even if we were to use the read REST endpoint, it would mean that the timeline client code would need to contact the list of all known version URLs one at a time to see which read endpoint is up and running. It sounds to me that it can be unnecessarily flaky and harder than it needs to be.

          I think it makes sense to rely on the version config, and not worry about querying it live. I don't see added value, and config will only be simpler and more robust. Have I missed any important point?

          Show
          sjlee0 Sangjin Lee added a comment - 4. Can raise new jira and handle REST interface support to get the supported ATS version and config from the server for ATS 1.5 5. Can raise new jira and handle version checking for the ATSv2 interface methods in TimelineClient.and also fetching of version In my mind, trying to get the version by querying the REST interface seems highly problematic. If we're going to have a configuration that clearly spells out which version the cluster is running, why not rely on that config? Is there a reason to believe that may not be correct? In terms of issues of querying the REST interface for the version, first, the write URLs are not really feasible. V.1.5 may not even have the write REST endpoint up. For v.2, you really need to first discover the right URL (i.e. server) to talk to. You do it anyway, but that's only to know which endpoint to talk to, not to discover which version it needs to use. Even if we were to use the read REST endpoint, it would mean that the timeline client code would need to contact the list of all known version URLs one at a time to see which read endpoint is up and running. It sounds to me that it can be unnecessarily flaky and harder than it needs to be. I think it makes sense to rely on the version config, and not worry about querying it live. I don't see added value, and config will only be simpler and more robust. Have I missed any important point?
          Hide
          djp Junping Du added a comment -

          I don't quite familiar with requirement of ATS v1.5. However, in stands of ATS v2, I would agree with Sangjin Lee's comments above to make this effect works on writer side only (TimelineClient).
          More clarifications:
          1. This version configuration is to benefit application/framework to have flexibility to run on top of YARN cluster with ATS v1 or v2 running with indicating the latest stable version ATS service that the cluster can support. ATS v1 and v2 client are different binary bits and use different incompatible APIs to put information like event, metrics, etc. so far. With getting proper configuration from YARN, the application can aware the ATS service version when landing on YARN cluster and can choose different TimelineClient to push info and get rid of our pains in doing TestDistributedCache for v1/v2 timeline service.

          2. We shouldn't break rolling upgrade scenario, or it could be seen as incompatible feature which cannot land on 2.x branch. That also means, we should support ATS v1 and v2 services at the same time during cluster upgrade so legacy/existing applications can still access their old ATS service which is the same as many rollup stories.

          2 clarification is more related to this change: we'd better change "yarn.timeline-service.version" to "yarn.timeline-service.latest.version" and use "indicate to clients what is the latest stable version of the running timeline service" to get rid of any confusion here. Also it is better to explicitly mention that our support range for ATS is: [X-1, X] for rolling upgrade (assume X is latest stable ATS version).

          Show
          djp Junping Du added a comment - I don't quite familiar with requirement of ATS v1.5. However, in stands of ATS v2, I would agree with Sangjin Lee 's comments above to make this effect works on writer side only (TimelineClient). More clarifications: 1. This version configuration is to benefit application/framework to have flexibility to run on top of YARN cluster with ATS v1 or v2 running with indicating the latest stable version ATS service that the cluster can support. ATS v1 and v2 client are different binary bits and use different incompatible APIs to put information like event, metrics, etc. so far. With getting proper configuration from YARN, the application can aware the ATS service version when landing on YARN cluster and can choose different TimelineClient to push info and get rid of our pains in doing TestDistributedCache for v1/v2 timeline service. 2. We shouldn't break rolling upgrade scenario, or it could be seen as incompatible feature which cannot land on 2.x branch. That also means, we should support ATS v1 and v2 services at the same time during cluster upgrade so legacy/existing applications can still access their old ATS service which is the same as many rollup stories. 2 clarification is more related to this change: we'd better change "yarn.timeline-service.version" to "yarn.timeline-service.latest.version" and use "indicate to clients what is the latest stable version of the running timeline service" to get rid of any confusion here. Also it is better to explicitly mention that our support range for ATS is: [X-1, X] for rolling upgrade (assume X is latest stable ATS version).
          Hide
          gtCarrera9 Li Lu added a comment -

          Thanks for the review Junping Du! The ATS v1.5 introduces some new API on top of ATS v1 APIs. However, ATS v2 is not compatible with either versions. I agree that a config would suffice to specify the "active" ATS version or the version of the writer API a client should use. Right now I think a config with name "yarn.timeline-service.version" is fine because this leaves flexibility to allow a set of active ATS writer API versions in the system. Marking a latest version may not be quite useful since ATS 1.x is not API-compatible with ATS v2.x.

          On the other hand, I totally agree there should be a comprehensive story for ATS rolling upgrade. IIUC, ATS v1 can be upgraded in a rolling fashion to v1.5. Meanwhile, if the ATS v1/1.5 server is available in the system, v1.x server should be able to work with v2.x clients (since the v1 server won't be touched by ATS v2 client). Therefore, I think the rolling upgrade story, from ATS v1.x to ATS v2, can be reduced to the ability for ATS v1 servers and ATS v2 can co-exist in the cluster? We can certainly have more discussion on the rolling upgrade in ATS v2 JIRAs.

          Show
          gtCarrera9 Li Lu added a comment - Thanks for the review Junping Du ! The ATS v1.5 introduces some new API on top of ATS v1 APIs. However, ATS v2 is not compatible with either versions. I agree that a config would suffice to specify the "active" ATS version or the version of the writer API a client should use. Right now I think a config with name "yarn.timeline-service.version" is fine because this leaves flexibility to allow a set of active ATS writer API versions in the system. Marking a latest version may not be quite useful since ATS 1.x is not API-compatible with ATS v2.x. On the other hand, I totally agree there should be a comprehensive story for ATS rolling upgrade. IIUC, ATS v1 can be upgraded in a rolling fashion to v1.5. Meanwhile, if the ATS v1/1.5 server is available in the system, v1.x server should be able to work with v2.x clients (since the v1 server won't be touched by ATS v2 client). Therefore, I think the rolling upgrade story, from ATS v1.x to ATS v2, can be reduced to the ability for ATS v1 servers and ATS v2 can co-exist in the cluster? We can certainly have more discussion on the rolling upgrade in ATS v2 JIRAs.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I agree the rolling upgrade use case from v.1 to v.2 should be addressed. We had some offline discussion on this too. Since it is a pretty major item in and of itself and somewhat separate (being v.2-specific) from this specific JIRA, it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good?

          I'm fine with the current name "yarn.timeline-service.version". I just want to clarify the interpretation of this config on the cluster side and on the client side.

          On the cluster side, it should always be interpreted as precisely which version of the timeline service should be up. If "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it should be understood as the cluster should bring up the timeline service v.1.5 (and nothing else), and the client can expect that to be the case.

          On the client side, clearly a client that uses the same version should expect to succeed. If a client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary (part of the rolling upgrade discussion included).

          Show
          sjlee0 Sangjin Lee added a comment - I agree the rolling upgrade use case from v.1 to v.2 should be addressed. We had some offline discussion on this too. Since it is a pretty major item in and of itself and somewhat separate (being v.2-specific) from this specific JIRA, it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good? I'm fine with the current name "yarn.timeline-service.version". I just want to clarify the interpretation of this config on the cluster side and on the client side. On the cluster side, it should always be interpreted as precisely which version of the timeline service should be up. If "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it should be understood as the cluster should bring up the timeline service v.1.5 (and nothing else), and the client can expect that to be the case. On the client side, clearly a client that uses the same version should expect to succeed. If a client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary (part of the rolling upgrade discussion included).
          Hide
          gtCarrera9 Li Lu added a comment -

          it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good?

          Agree. We can further investigate this issue in the v2 branch. I'm also fine with the current config name in Xuan Gong's patch.

          Show
          gtCarrera9 Li Lu added a comment - it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good? Agree. We can further investigate this issue in the v2 branch. I'm also fine with the current config name in Xuan Gong 's patch.
          Hide
          djp Junping Du added a comment -

          I agree the rolling upgrade use case from v.1 to v.2 should be addressed. We had some offline discussion on this too. Since it is a pretty major item in and of itself and somewhat separate (being v.2-specific) from this specific JIRA, it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good?

          Sounds good.

          I'm fine with the current name "yarn.timeline-service.version". I just want to clarify the interpretation of this config on the cluster side and on the client side.

          If all of you are fine with current config name, I would compromise on this. Yes. More clarification is needed on this configuration so we should definitely update the description.

          On the cluster side, it should always be interpreted as precisely which version of the timeline service should be up. If "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it should be understood as the cluster should bring up the timeline service v.1.5 (and nothing else), and the client can expect that to be the case.

          I would question on this. If "yarn.timeline-service.version" is 2 (after cluster is being upgrade), and we don't serve 1/1.5 ATS service any more, how can existing running applications survival for timeline services? Unless we have a clear answer in v2 that we will continue to maintain a ATS v1/v1.5 service as a legacy daemon in v2 (I don't prefer this way), I don't think we should mark this config to indicate an unique version of ATS service running in the server side.

          On the client side, clearly a client that uses the same version should expect to succeed. If a client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary (part of the rolling upgrade discussion included).

          This works if we don't consider rolling upgrade case. For roll up cases, an running application/framework cannot switch its client version config if YARN cluster is upgrading to a new version ATS. We shouldn't claim that application's clients is expected to be no response if version is mis-match with serve or the user would misunderstand they have to kill these applications after upgrade. Instead, we should claim that client is not supposed to override this config that vary with cluster config unless they are pretty sure what cluster side are doing (like upgrading process, etc.).

          Show
          djp Junping Du added a comment - I agree the rolling upgrade use case from v.1 to v.2 should be addressed. We had some offline discussion on this too. Since it is a pretty major item in and of itself and somewhat separate (being v.2-specific) from this specific JIRA, it may be sufficient to note it here and carry on that discussion on a v.2 subtask. Sound good? Sounds good. I'm fine with the current name "yarn.timeline-service.version". I just want to clarify the interpretation of this config on the cluster side and on the client side. If all of you are fine with current config name, I would compromise on this. Yes. More clarification is needed on this configuration so we should definitely update the description. On the cluster side, it should always be interpreted as precisely which version of the timeline service should be up. If "yarn.timeline-service.version" is 1.5, and "yarn.timeline-service.enabled" is true, it should be understood as the cluster should bring up the timeline service v.1.5 (and nothing else), and the client can expect that to be the case. I would question on this. If "yarn.timeline-service.version" is 2 (after cluster is being upgrade), and we don't serve 1/1.5 ATS service any more, how can existing running applications survival for timeline services? Unless we have a clear answer in v2 that we will continue to maintain a ATS v1/v1.5 service as a legacy daemon in v2 (I don't prefer this way), I don't think we should mark this config to indicate an unique version of ATS service running in the server side. On the client side, clearly a client that uses the same version should expect to succeed. If a client chooses to use a smaller version in spite of this, then depending on how robust the compatibility story is between versions, the results may vary (part of the rolling upgrade discussion included). This works if we don't consider rolling upgrade case. For roll up cases, an running application/framework cannot switch its client version config if YARN cluster is upgrading to a new version ATS. We shouldn't claim that application's clients is expected to be no response if version is mis-match with serve or the user would misunderstand they have to kill these applications after upgrade. Instead, we should claim that client is not supposed to override this config that vary with cluster config unless they are pretty sure what cluster side are doing (like upgrading process, etc.).
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for your comment Junping Du. Per comments above, let's move the v.1-v.2 compatibility discussion to YARN-3196. I just want to clarify my comments to the extent it is relevant to this JIRA as I think they might have been misunderstood.

          I would question on this. If "yarn.timeline-service.version" is 2 (after cluster is being upgrade), and we don't serve 1/1.5 ATS service any more, how can existing running applications survival for timeline services? Unless we have a clear answer in v2 that we will continue to maintain a ATS v1/v1.5 service as a legacy daemon in v2 (I don't prefer this way), I don't think we should mark this config to indicate an unique version of ATS service running in the server side.

          When I said the cluster should bring up that exact version of the timeline service, I didn't mean that we will not support any compatibility. I definitely agree that the compatibility and support for a smooth rolling upgrade should be an objective, and that's why we want to continue the discussion and work on YARN-3196. What I meant to do is to separate the compatibility support (or rolling upgrade support) from the main interpretation of this config on the cluster.

          It is true that the main mode of operation will be on the version that is declared via timeline-service.version. Also, I think there are many options in the way we can implement the rolling upgrade support. Supporting rolling upgrade does not necessarily mean that the v.1 write/read endpoints must be up in parallel with the v.2 write/read endpoints. We talked about having some kind of a temporary proxy or something in the timeline client itself. There may be other ways, but we're not mandating that the old endpoints must be up to implement the rolling upgrade support. My point was that when we say timeline-service.version = 2 doesn't automatically mean we still must bring up end points of the previous version (or versions), as that's more of an implementation choice for how to support rolling upgrade. I hope that clarifies my earlier comments.

          This works if we don't consider rolling upgrade case. For roll up cases, an running application/framework cannot switch its client version config if YARN cluster is upgrading to a new version ATS. We shouldn't claim that application's clients is expected to be no response if version is mis-match with serve or the user would misunderstand they have to kill these applications after upgrade. Instead, we should claim that client is not supposed to override this config that vary with cluster config unless they are pretty sure what cluster side are doing (like upgrading process, etc.).

          Again, I hope it is clear what I meant was NOT that we will not consider the rolling upgrade use case. Even if the cluster is running with version = 2, with a proper rolling upgrade support, it should be prepared to handle (during the transition) calls that are coming in from running apps with version = 1 or 1.5. That's why I said "depending on how robust the compatibility story is".

          Let me know if this helps in any way.

          Show
          sjlee0 Sangjin Lee added a comment - Thanks for your comment Junping Du . Per comments above, let's move the v.1-v.2 compatibility discussion to YARN-3196 . I just want to clarify my comments to the extent it is relevant to this JIRA as I think they might have been misunderstood. I would question on this. If "yarn.timeline-service.version" is 2 (after cluster is being upgrade), and we don't serve 1/1.5 ATS service any more, how can existing running applications survival for timeline services? Unless we have a clear answer in v2 that we will continue to maintain a ATS v1/v1.5 service as a legacy daemon in v2 (I don't prefer this way), I don't think we should mark this config to indicate an unique version of ATS service running in the server side. When I said the cluster should bring up that exact version of the timeline service, I didn't mean that we will not support any compatibility. I definitely agree that the compatibility and support for a smooth rolling upgrade should be an objective, and that's why we want to continue the discussion and work on YARN-3196 . What I meant to do is to separate the compatibility support (or rolling upgrade support) from the main interpretation of this config on the cluster. It is true that the main mode of operation will be on the version that is declared via timeline-service.version. Also, I think there are many options in the way we can implement the rolling upgrade support. Supporting rolling upgrade does not necessarily mean that the v.1 write/read endpoints must be up in parallel with the v.2 write/read endpoints. We talked about having some kind of a temporary proxy or something in the timeline client itself. There may be other ways, but we're not mandating that the old endpoints must be up to implement the rolling upgrade support. My point was that when we say timeline-service.version = 2 doesn't automatically mean we still must bring up end points of the previous version (or versions), as that's more of an implementation choice for how to support rolling upgrade. I hope that clarifies my earlier comments. This works if we don't consider rolling upgrade case. For roll up cases, an running application/framework cannot switch its client version config if YARN cluster is upgrading to a new version ATS. We shouldn't claim that application's clients is expected to be no response if version is mis-match with serve or the user would misunderstand they have to kill these applications after upgrade. Instead, we should claim that client is not supposed to override this config that vary with cluster config unless they are pretty sure what cluster side are doing (like upgrading process, etc.). Again, I hope it is clear what I meant was NOT that we will not consider the rolling upgrade use case. Even if the cluster is running with version = 2, with a proper rolling upgrade support, it should be prepared to handle (during the transition) calls that are coming in from running apps with version = 1 or 1.5. That's why I said "depending on how robust the compatibility story is". Let me know if this helps in any way.
          Hide
          gtCarrera9 Li Lu added a comment -

          Hi folks, seems like we're reaching agreement on the config itself, but still having some concerns with timeline v2 rolling upgrade. How about fix the description as Junping suggested, put this patch in while we keep our v2 discussion in YARN-3196? Looks like the only action item for this JIRA is to update the description?

          Show
          gtCarrera9 Li Lu added a comment - Hi folks, seems like we're reaching agreement on the config itself, but still having some concerns with timeline v2 rolling upgrade. How about fix the description as Junping suggested, put this patch in while we keep our v2 discussion in YARN-3196 ? Looks like the only action item for this JIRA is to update the description?
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'm +1. Thanks.

          Show
          sjlee0 Sangjin Lee added a comment - I'm +1. Thanks.
          Hide
          xgong Xuan Gong added a comment -

          fix the description.

          Show
          xgong Xuan Gong added a comment - fix the description.
          Hide
          gtCarrera9 Li Lu added a comment -

          The new description LGTM. Sangjin Lee any suggestions? Thanks!

          Show
          gtCarrera9 Li Lu added a comment - The new description LGTM. Sangjin Lee any suggestions? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 33s trunk passed
          +1 compile 2m 20s trunk passed with JDK v1.8.0_66
          +1 compile 2m 20s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 46s trunk passed
          +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66
          +1 javadoc 3m 25s trunk passed with JDK v1.7.0_91
          +1 mvninstall 1m 0s the patch passed
          +1 compile 2m 3s the patch passed with JDK v1.8.0_66
          +1 javac 2m 3s the patch passed
          +1 compile 2m 18s the patch passed with JDK v1.7.0_91
          +1 javac 2m 18s the patch passed
          -1 checkstyle 0m 29s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 214, now 214).
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 26s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 7s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 3m 24s the patch passed with JDK v1.7.0_91
          +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66.
          +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66.
          +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91.
          +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_91.
          +1 asflicense 0m 23s Patch does not generate ASF License warnings.
          44m 28s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776639/YARN-3623-2015-12-09.patch
          JIRA Issue YARN-3623
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 98c6886eaced 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 50edcb9
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9916/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9916/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn
          Max memory used 76MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9916/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 33s trunk passed +1 compile 2m 20s trunk passed with JDK v1.8.0_66 +1 compile 2m 20s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 46s trunk passed +1 javadoc 1m 10s trunk passed with JDK v1.8.0_66 +1 javadoc 3m 25s trunk passed with JDK v1.7.0_91 +1 mvninstall 1m 0s the patch passed +1 compile 2m 3s the patch passed with JDK v1.8.0_66 +1 javac 2m 3s the patch passed +1 compile 2m 18s the patch passed with JDK v1.7.0_91 +1 javac 2m 18s the patch passed -1 checkstyle 0m 29s Patch generated 1 new checkstyle issues in hadoop-yarn-project/hadoop-yarn (total was 214, now 214). +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 26s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 7s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 3m 24s the patch passed with JDK v1.7.0_91 +1 unit 0m 24s hadoop-yarn-api in the patch passed with JDK v1.8.0_66. +1 unit 2m 5s hadoop-yarn-common in the patch passed with JDK v1.8.0_66. +1 unit 0m 26s hadoop-yarn-api in the patch passed with JDK v1.7.0_91. +1 unit 2m 16s hadoop-yarn-common in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 44m 28s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12776639/YARN-3623-2015-12-09.patch JIRA Issue YARN-3623 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 98c6886eaced 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 50edcb9 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/9916/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9916/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn Max memory used 76MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-YARN-Build/9916/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'm obviously +1 as that's exactly what I said. Junping Du?

          Show
          sjlee0 Sangjin Lee added a comment - I'm obviously +1 as that's exactly what I said. Junping Du ?
          Hide
          djp Junping Du added a comment -

          +1. Committing.

          Show
          djp Junping Du added a comment - +1. Committing.
          Hide
          djp Junping Du added a comment -

          Oh. Just noticed target version is YARN-2928 but I think ATS 1.5 need this also. Isn't it? Shall we commit the patch to trunk/branch-2?

          Show
          djp Junping Du added a comment - Oh. Just noticed target version is YARN-2928 but I think ATS 1.5 need this also. Isn't it? Shall we commit the patch to trunk/branch-2?
          Hide
          djp Junping Du added a comment -

          Ok. I think I already find the answer from YARN-4234 that ATS 1.5 depends on this patch. Will go ahead to commit it to 2.8.

          Show
          djp Junping Du added a comment - Ok. I think I already find the answer from YARN-4234 that ATS 1.5 depends on this patch. Will go ahead to commit it to 2.8.
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Xuan Gong for delivering the patch, and Sangjin, Li and Naga for review and comments!

          Show
          djp Junping Du added a comment - I have commit the patch to trunk, branch-2 and branch-2.8. Thanks Xuan Gong for delivering the patch, and Sangjin, Li and Naga for review and comments!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8950 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8950/)
          YARN-3623. Add a config to indicate the Timeline Service version. (junping_du: rev f910e4f639dc311fcb257bfcb869b1aa8b2c0643)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8950 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8950/ ) YARN-3623 . Add a config to indicate the Timeline Service version. (junping_du: rev f910e4f639dc311fcb257bfcb869b1aa8b2c0643) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #682 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/682/)
          YARN-3623. Add a config to indicate the Timeline Service version. (junping_du: rev f910e4f639dc311fcb257bfcb869b1aa8b2c0643)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #682 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/682/ ) YARN-3623 . Add a config to indicate the Timeline Service version. (junping_du: rev f910e4f639dc311fcb257bfcb869b1aa8b2c0643) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt
          Hide
          djp Junping Du added a comment -

          "it means the cluster will and should bring up the timeline service v.1.5 (and nothing else)."

          Think over it again. I am still uncomfortable for the description - "(and nothing else)." It could be against for our possible solutions for YARN-4368. I prefer to remove this in an addendum patch or it could provide unnecessary restrictions if it get released in 2.8.0. Thoughts?

          Show
          djp Junping Du added a comment - "it means the cluster will and should bring up the timeline service v.1.5 (and nothing else)." Think over it again. I am still uncomfortable for the description - "(and nothing else)." It could be against for our possible solutions for YARN-4368 . I prefer to remove this in an addendum patch or it could provide unnecessary restrictions if it get released in 2.8.0. Thoughts?
          Hide
          sjlee0 Sangjin Lee added a comment -

          That's why I asked whether you were OK with the wording. How about "it means the cluster should bring up the timeline service specified by the version" (dropping the exclusive phrase)?

          Show
          sjlee0 Sangjin Lee added a comment - That's why I asked whether you were OK with the wording. How about "it means the cluster should bring up the timeline service specified by the version" (dropping the exclusive phrase)?
          Hide
          djp Junping Du added a comment -

          Sounds good. I put an addendum patch here. Please check if it make sense.

          Show
          djp Junping Du added a comment - Sounds good. I put an addendum patch here. Please check if it make sense.
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          +1 lgtm

          Show
          Naganarasimha Naganarasimha G R added a comment - +1 lgtm
          Hide
          gtCarrera9 Li Lu added a comment -

          I'd like to make sure I understand this change correctly. Specifically, what kind of new cases do we allow by removing the "nothing else" part. IIUC, this allows us to bring up an old ATS server after the system has been upgraded. This looks fine with me. Am I missing some other cases introduced by this change?

          Show
          gtCarrera9 Li Lu added a comment - I'd like to make sure I understand this change correctly. Specifically, what kind of new cases do we allow by removing the "nothing else" part. IIUC, this allows us to bring up an old ATS server after the system has been upgraded. This looks fine with me. Am I missing some other cases introduced by this change?
          Hide
          djp Junping Du added a comment -

          Yes. That's the flexibility we want here - we probably bring up old version ATS service for legacy running apps during upgrade. So we should pay more attentions here given configurations (especially in yarn-default.xml) serve as a public protocol to hadoop users.

          Show
          djp Junping Du added a comment - Yes. That's the flexibility we want here - we probably bring up old version ATS service for legacy running apps during upgrade. So we should pay more attentions here given configurations (especially in yarn-default.xml) serve as a public protocol to hadoop users.
          Hide
          gtCarrera9 Li Lu added a comment -

          Agree. +1 for the change.

          Show
          gtCarrera9 Li Lu added a comment - Agree. +1 for the change.
          Hide
          xgong Xuan Gong added a comment -

          +1 Checking this in

          Show
          xgong Xuan Gong added a comment - +1 Checking this in
          Hide
          xgong Xuan Gong added a comment -

          Committed the addendum patch in trunk/branch-2. Thanks, junping.

          Show
          xgong Xuan Gong added a comment - Committed the addendum patch in trunk/branch-2. Thanks, junping.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8951 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8951/)
          YARN-3623-Addendum: Improve the description for Timeline Service Version (xgong: rev 21daa6c68a0bff51a14e748bf14d56b2f5a5580f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8951 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8951/ ) YARN-3623 -Addendum: Improve the description for Timeline Service Version (xgong: rev 21daa6c68a0bff51a14e748bf14d56b2f5a5580f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/683/)
          YARN-3623-Addendum: Improve the description for Timeline Service Version (xgong: rev 21daa6c68a0bff51a14e748bf14d56b2f5a5580f)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Hdfs-trunk-Java8 #683 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/683/ ) YARN-3623 -Addendum: Improve the description for Timeline Service Version (xgong: rev 21daa6c68a0bff51a14e748bf14d56b2f5a5580f) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10074/)
          YARN-3623. Add a config to indicate the Timeline Service version. (sjlee: rev cf4666bb8ccc99a63c623455cd88157a3007775d)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10074 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10074/ ) YARN-3623 . Add a config to indicate the Timeline Service version. (sjlee: rev cf4666bb8ccc99a63c623455cd88157a3007775d) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml

            People

            • Assignee:
              xgong Xuan Gong
              Reporter:
              zjshen Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development