Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1700

Package YARN Timeline Server service for Apache Hadoop

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Apache Hadoop (>= 2.4.0) includes an YARN timeline server service. http://hadoop.apache.org/docs/r2.4.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html

      1. BIGTOP-1700.1.patch
        9 kB
        YoungWoo Kim
      2. BIGTOP-1700.2.patch
        6 kB
        YoungWoo Kim
      3. BIGTOP-1700.3.patch
        9 kB
        YoungWoo Kim

        Issue Links

          Activity

          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Attached patch BIGTOP-1700.1.patch

          # service hadoop-yarn-timelineserver start
          
          

          Browse the UI for yarn timelineserver, http://HOSTNAME:8188/

          Show
          warwithin YoungWoo Kim added a comment - - edited Attached patch BIGTOP-1700.1.patch # service hadoop-yarn-timelineserver start Browse the UI for yarn timelineserver, http://HOSTNAME:8188/
          Hide
          warwithin YoungWoo Kim added a comment -

          Updated patch, BIGTOP-1700.2.patch

          Show
          warwithin YoungWoo Kim added a comment - Updated patch, BIGTOP-1700.2.patch
          Hide
          evans_ye Evans Ye added a comment -

          LGTM. Just one question: why DAEMON_FLAGS="historyserver"?

          Show
          evans_ye Evans Ye added a comment - LGTM. Just one question: why DAEMON_FLAGS="historyserver"?
          Hide
          warwithin YoungWoo Kim added a comment - - edited

          Evans Ye,
          http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html:

          Storage and retrieval of applications' current as well as historic information in a generic fashion is solved in YARN through the Timeline Server (previously also called Generic Application History Server).

          See also, YARN-1530

          Right now, It's called 'yarn timelineserver' but the command and args are not changed from AHS

          Show
          warwithin YoungWoo Kim added a comment - - edited Evans Ye , http://hadoop.apache.org/docs/r2.6.0/hadoop-yarn/hadoop-yarn-site/TimelineServer.html: Storage and retrieval of applications' current as well as historic information in a generic fashion is solved in YARN through the Timeline Server (previously also called Generic Application History Server). See also, YARN-1530 Right now, It's called 'yarn timelineserver' but the command and args are not changed from AHS
          Hide
          jayunit100 jay vyas added a comment - - edited

          YoungWoo Kim since there is a easy test for this, i assume a wget or something , can we have a smoke test for this?

          its quite easy to add new smoke tests, just a groovy script that you can dump in https://github.com/apache/bigtop/tree/master/bigtop-tests/smoke-tests (add a yarn directrory). then you can run it easily by just ./gradlew compileGroovy test -Dsmoke.tests=yarn --info .

          Either it can be a part of this patch or a later jira ... thanks !

          Show
          jayunit100 jay vyas added a comment - - edited YoungWoo Kim since there is a easy test for this, i assume a wget or something , can we have a smoke test for this? its quite easy to add new smoke tests, just a groovy script that you can dump in https://github.com/apache/bigtop/tree/master/bigtop-tests/smoke-tests (add a yarn directrory). then you can run it easily by just ./gradlew compileGroovy test -Dsmoke.tests=yarn --info . Either it can be a part of this patch or a later jira ... thanks !
          Hide
          warwithin YoungWoo Kim added a comment -

          jay vyas I see. Filed a jira for the test, BIGTOP-1714. Thanks!

          Show
          warwithin YoungWoo Kim added a comment - jay vyas I see. Filed a jira for the test, BIGTOP-1714 . Thanks!
          Hide
          jayunit100 jay vyas added a comment -

          Great ! Evans Ye ill let you finish this review off .

          Show
          jayunit100 jay vyas added a comment - Great ! Evans Ye ill let you finish this review off .
          Hide
          mgrover Mark Grover added a comment -

          I am reviewing too, btw. Thanks for this!

          Show
          mgrover Mark Grover added a comment - I am reviewing too, btw. Thanks for this!
          Hide
          evans_ye Evans Ye added a comment -

          Hi Mark Grover please go ahead. It works for me, but I don't have to much knowledge on this. Just one more thing, YoungWoo Kim could you please create another ticket for timeline server deployment in puppet recipes?

          Show
          evans_ye Evans Ye added a comment - Hi Mark Grover please go ahead. It works for me, but I don't have to much knowledge on this. Just one more thing, YoungWoo Kim could you please create another ticket for timeline server deployment in puppet recipes?
          Hide
          mgrover Mark Grover added a comment -

          +1 from me too. Thanks. In full disclosure, I haven't tried starting the service on various OSs. I am assuming that's been done and tested already?

          As far as I know, YARN is working on a new ATS (see YARN-2928) so when that comes, it will probably subsume this.

          Show
          mgrover Mark Grover added a comment - +1 from me too. Thanks. In full disclosure, I haven't tried starting the service on various OSs. I am assuming that's been done and tested already? As far as I know, YARN is working on a new ATS (see YARN-2928 ) so when that comes, it will probably subsume this.
          Hide
          warwithin YoungWoo Kim added a comment -

          Mark Grover Thanks for your comment! I've just checked availability of the service actually. I believe, there are some work to do... Integration into Bigtop, configuration changes, smoke tests and so on.

          Show
          warwithin YoungWoo Kim added a comment - Mark Grover Thanks for your comment! I've just checked availability of the service actually. I believe, there are some work to do... Integration into Bigtop, configuration changes, smoke tests and so on.
          Hide
          warwithin YoungWoo Kim added a comment -

          I found an issue from the latest patch. I'll update a new patch.

          Show
          warwithin YoungWoo Kim added a comment - I found an issue from the latest patch. I'll update a new patch.
          Hide
          warwithin YoungWoo Kim added a comment -

          BIGTOP-1700.3.patch

          • Change 'daemon flag'

          Now, YARN Timeline server is 'timelineserver' on Hadoop 2.6.

          Show
          warwithin YoungWoo Kim added a comment - BIGTOP-1700.3.patch Change 'daemon flag' Now, YARN Timeline server is 'timelineserver' on Hadoop 2.6.
          Hide
          mgrover Mark Grover added a comment -

          LGTM +1

          Show
          mgrover Mark Grover added a comment - LGTM +1
          Hide
          mgrover Mark Grover added a comment -

          I will give it a couple more hours for others to review it, otherwise will commit this afternoon.

          Show
          mgrover Mark Grover added a comment - I will give it a couple more hours for others to review it, otherwise will commit this afternoon.
          Hide
          mgrover Mark Grover added a comment -

          And, committed! Thanks YoungWoo!

          Show
          mgrover Mark Grover added a comment - And, committed! Thanks YoungWoo!
          Hide
          warwithin YoungWoo Kim added a comment -

          Mark Grover, Evans Ye.

          Thanks for reviewing the patch!

          Show
          warwithin YoungWoo Kim added a comment - Mark Grover , Evans Ye . Thanks for reviewing the patch!

            People

            • Assignee:
              warwithin YoungWoo Kim
              Reporter:
              warwithin YoungWoo Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development