Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-6018.patch
      10 kB
      Robert Kanter
    2. MAPREDUCE-6018.patch
      10 kB
      Robert Kanter
    3. MAPREDUCE-6018.patch
      10 kB
      Zhijie Shen

      Issue Links

        Activity

        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1915/)
        MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1915 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1915/ ) MAPREDUCE-6018 . Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1940 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1940/)
        MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63)

        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1940 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1940/ ) MAPREDUCE-6018 . Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #726 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/726/)
        MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #726 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/726/ ) MAPREDUCE-6018 . Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #6363 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6363/)
        MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #6363 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6363/ ) MAPREDUCE-6018 . Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. (zjshen: rev 971e91c8c03a23e4613ed3f071b4f982ee5a1b63) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        Zhijie Shen added a comment -

        Committed the patch to trunk, branch-2 and branch-2.6. Thanks Robert for the patch, and Vinod for input about config name!

        Show
        Zhijie Shen added a comment - Committed the patch to trunk, branch-2 and branch-2.6. Thanks Robert for the patch, and Vinod for input about config name!
        Hide
        Zhijie Shen added a comment -

        Again, Jenkins is temporally broken. The last patch is minor change upon the prior one. Will go ahead to commit it directly.

        Show
        Zhijie Shen added a comment - Again, Jenkins is temporally broken. The last patch is minor change upon the prior one. Will go ahead to commit it directly.
        Hide
        Zhijie Shen added a comment -

        +1 for the last patch. Do some cosmetic work: fix the config description and add a log. Will commit this patch after MAPREDUCE-5933

        Show
        Zhijie Shen added a comment - +1 for the last patch. Do some cosmetic work: fix the config description and add a log. Will commit this patch after MAPREDUCE-5933
        Hide
        Robert Kanter added a comment -

        The new patch gets rid of the boolean, renames the config property to mapreduce.job.emit-timeline-data (which I think is more clear), defaults it to false, and adds it to mapped-site.xml.

        Show
        Robert Kanter added a comment - The new patch gets rid of the boolean, renames the config property to mapreduce.job.emit-timeline-data (which I think is more clear), defaults it to false , and adds it to mapped-site.xml.
        Hide
        Zhijie Shen added a comment -

        mapreduce.job.emit-timeline-data

        Thanks for clarification. I used to think "emit-timeline-data" means a sub module of job. In MAPREDUCE-5933, we actually didn't introduce an obvious submodule, so let's use mapreduce.job.emit-timeline-data = false.

        Show
        Zhijie Shen added a comment - mapreduce.job.emit-timeline-data Thanks for clarification. I used to think "emit-timeline-data" means a sub module of job. In MAPREDUCE-5933 , we actually didn't introduce an obvious submodule, so let's use mapreduce.job.emit-timeline-data = false.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        mapreduce.job.emit-timeline-data is already a boolean, no need for enabled. Otherwise we can follow RM's config and say mapreduce.job.timeline-data-publisher.enabled?

        Show
        Vinod Kumar Vavilapalli added a comment - mapreduce.job.emit-timeline-data is already a boolean, no need for enabled. Otherwise we can follow RM's config and say mapreduce.job.timeline-data-publisher.enabled ?
        Hide
        Zhijie Shen added a comment -

        Let's say mapreduce.job.emit-timeline-data.enabled?

        Show
        Zhijie Shen added a comment - Let's say mapreduce.job.emit-timeline-data.enabled?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Haven't looked at all of the patch, but one comment: How about mapreduce.job.emit-timeline-data (or something similar) instead of mapreduce.job.timeline-service.enabled? If you agree, similarly rename the constants.

        Show
        Vinod Kumar Vavilapalli added a comment - Haven't looked at all of the patch, but one comment: How about mapreduce.job.emit-timeline-data (or something similar) instead of mapreduce.job.timeline-service.enabled ? If you agree, similarly rename the constants.
        Hide
        Zhijie Shen added a comment -

        I forgot mentioning how about using false as the default value? MR is a long-live and widely used framework, let's be more conservative when introducing a new feature it.

        Show
        Zhijie Shen added a comment - I forgot mentioning how about using false as the default value? MR is a long-live and widely used framework, let's be more conservative when introducing a new feature it.
        Hide
        Zhijie Shen added a comment -

        Thanks for the patch, Robert Kanter! Some minor comments about it.

        1. It's not necessary var. The client is only constructed when the flag is set. Then we try to use timeline client, just check null first.

        +  private boolean timelineEnabled;
        

        2. Can you add the config to mapred-default.xml and add a description?

        Show
        Zhijie Shen added a comment - Thanks for the patch, Robert Kanter ! Some minor comments about it. 1. It's not necessary var. The client is only constructed when the flag is set. Then we try to use timeline client, just check null first. + private boolean timelineEnabled; 2. Can you add the config to mapred-default.xml and add a description?
        Hide
        Robert Kanter added a comment -

        I added the config to MRJobConfig and named it mapreduce.job.timeline-service.enabled and defaults to true.

        This requires MAPREDUCE-5933.

        Show
        Robert Kanter added a comment - I added the config to MRJobConfig and named it mapreduce.job.timeline-service.enabled and defaults to true. This requires MAPREDUCE-5933 .
        Hide
        Zhijie Shen added a comment -

        Robert Kanter, thanks for taking care of this issue. We could have a MR framework-level config, such as "mapreduce.jobhistory.timeline-service.enabled", and it can also be overwritten by each individual MR job by specifying it in CLI args. Only when this flag is set, the logic that history events are posted to the timeline server will be executed.

        Show
        Zhijie Shen added a comment - Robert Kanter , thanks for taking care of this issue. We could have a MR framework-level config, such as "mapreduce.jobhistory.timeline-service.enabled", and it can also be overwritten by each individual MR job by specifying it in CLI args. Only when this flag is set, the logic that history events are posted to the timeline server will be executed.

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development