Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5858

[Umbrella] MR should make use of the timeline server

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Now MR relies on its own JobHistoryServer for MR specific history information. Given the timeline server is ready, we should gradually migrate MR historic data to it as well. relieving MR from maintaining its own history server daemon.

        Issue Links

          Activity

          Zhijie Shen created issue -
          Zhijie Shen made changes -
          Field Original Value New Value
          Link This issue depends upon YARN-1530 [ YARN-1530 ]
          Zhijie Shen made changes -
          Summary MR should make use of the timeline server [Umbrella] MR should make use of the timeline server
          Zhijie Shen made changes -
          Assignee Zhijie Shen [ zjshen ]
          Hide
          Zhijie Shen added a comment -

          Unassigned the umbrella ticket as it may be contributed by multiple stakeholders.

          Show
          Zhijie Shen added a comment - Unassigned the umbrella ticket as it may be contributed by multiple stakeholders.
          Hide
          Jonathan Eagles added a comment -

          Zhijie Shen, we need to be careful when designing this. In particular, users of MR job history server will want to:

          • Continue using Job history server with no runtime dependency a timeline server being present
          • Enable or disable timeline server from the client side
          • automatic or manual pluggable configuration or disabling of an in memory solution for local mode testing
          • minicluster integration
          • Allow for history enabled for timeline server and job history server at the same time
          • What to do for the tracking url for different scenarios above
          • New MR UI using backing timeline backing store
          Show
          Jonathan Eagles added a comment - Zhijie Shen , we need to be careful when designing this. In particular, users of MR job history server will want to: Continue using Job history server with no runtime dependency a timeline server being present Enable or disable timeline server from the client side automatic or manual pluggable configuration or disabling of an in memory solution for local mode testing minicluster integration Allow for history enabled for timeline server and job history server at the same time What to do for the tracking url for different scenarios above New MR UI using backing timeline backing store
          Hide
          Zhijie Shen added a comment -

          Jonathan Eagles, thanks for your feedback, which is useful suggestion. If you have an more detailed design doc, please share with the community.

          As JHS has lived for a long time, we should rebase it on the timeline server gradually. IMHO, the first step is to make the MR job not only write the history events to the history files on HDFS, but also publish them to the timeline server, and it is configurable whether the MR job should write history files or upload timeline data.

          Once this is done, we can make JHS have the option to source the timeline server for MR jobs' history information, while we preserve the web front. Again it is the configurable to choose the data source.

          Once the data layer is stable, we can further think about migrating the UI. We can either build web UI on the third-party tool (such as Ambari) or even create web UI plugin in the timeline server. Any thoughts?

          Show
          Zhijie Shen added a comment - Jonathan Eagles , thanks for your feedback, which is useful suggestion. If you have an more detailed design doc, please share with the community. As JHS has lived for a long time, we should rebase it on the timeline server gradually. IMHO, the first step is to make the MR job not only write the history events to the history files on HDFS, but also publish them to the timeline server, and it is configurable whether the MR job should write history files or upload timeline data. Once this is done, we can make JHS have the option to source the timeline server for MR jobs' history information, while we preserve the web front. Again it is the configurable to choose the data source. Once the data layer is stable, we can further think about migrating the UI. We can either build web UI on the third-party tool (such as Ambari) or even create web UI plugin in the timeline server. Any thoughts?
          Zhijie Shen made changes -
          Link This issue relates to MAPREDUCE-6331 [ MAPREDUCE-6331 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

              • Created:
                Updated:

                Development