Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We should add the JvmPauseMonitor from HADOOP-9618 to the Job History Server.

      1. MAPREDUCE-6443.002.patch
        2 kB
        Robert Kanter
      2. MAPREDUCE-6443.001.patch
        2 kB
        Robert Kanter

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          in HADOOP-12313 I've proposed making JvmPauseMonitor a subclass of AbstractService; this will make it trivial to add the monitor to any CompositeService, including JHS. While I think the pause monitor is a great idea, it needs to become part of the YARN service model to be avoid patching in something with a different lifecycle and failure modes.

          Show
          stevel@apache.org Steve Loughran added a comment - in HADOOP-12313 I've proposed making JvmPauseMonitor a subclass of AbstractService; this will make it trivial to add the monitor to any CompositeService, including JHS. While I think the pause monitor is a great idea, it needs to become part of the YARN service model to be avoid patching in something with a different lifecycle and failure modes.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2207 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2207/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #269 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/269/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1010 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1010/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #280 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/280/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2225 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2225/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2225 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2225/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/276/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #276 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/276/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          rkanter Robert Kanter added a comment -

          Thanks for the review Junping Du

          Show
          rkanter Robert Kanter added a comment - Thanks for the review Junping Du
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8272 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8272/)
          MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8272 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8272/ ) MAPREDUCE-6443 . Add JvmPauseMonitor to JobHistoryServer. Contributed by Robert Kanter. (junping_du: rev e73a928a6360f68aaee2ed58b3a8d180f4051407) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistoryServer.java
          Hide
          djp Junping Du added a comment -

          I have commit the patch to trunk and branch-2. Thanks Robert Kanter for the contribution!

          Show
          djp Junping Du added a comment - I have commit the patch to trunk and branch-2. Thanks Robert Kanter for the contribution!
          Hide
          djp Junping Du added a comment -

          +1. 002 patch LGTM. Will commit it shortly.

          Show
          djp Junping Du added a comment - +1. 002 patch LGTM. Will commit it shortly.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 15m 38s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 56s There were no new javac warning messages.
          +1 javadoc 10m 43s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 15s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 24s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 5m 55s Tests passed in hadoop-mapreduce-client-hs.
              43m 43s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748899/MAPREDUCE-6443.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 4ab49a4
          hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 15m 38s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 56s There were no new javac warning messages. +1 javadoc 10m 43s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 15s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 24s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 0m 52s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 5m 55s Tests passed in hadoop-mapreduce-client-hs.     43m 43s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748899/MAPREDUCE-6443.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 4ab49a4 hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5929/console This message was automatically generated.
          Hide
          rkanter Robert Kanter added a comment -

          The existing metrics init stuff was in the start method, so I moved them all the init method too, in the 002 patch.

          Show
          rkanter Robert Kanter added a comment - The existing metrics init stuff was in the start method, so I moved them all the init method too, in the 002 patch.
          Hide
          djp Junping Du added a comment -

          Thanks for the patch, Robert. A similar comment with YARN-4019: can we move initiate work of pauseMonitor to serviceInit() and only left start work at serviceStart()?

          Show
          djp Junping Du added a comment - Thanks for the patch, Robert. A similar comment with YARN-4019 : can we move initiate work of pauseMonitor to serviceInit() and only left start work at serviceStart()?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 16m 1s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 javac 7m 40s There were no new javac warning messages.
          +1 javadoc 9m 41s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 29s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 23s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 5m 47s Tests passed in hadoop-mapreduce-client-hs.
              42m 49s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12748751/MAPREDUCE-6443.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / c95993c
          hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 16m 1s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 40s There were no new javac warning messages. +1 javadoc 9m 41s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 29s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 23s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 5m 47s Tests passed in hadoop-mapreduce-client-hs.     42m 49s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12748751/MAPREDUCE-6443.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / c95993c hadoop-mapreduce-client-hs test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5928/console This message was automatically generated.

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development