Details

    • Hadoop Flags:
      Reviewed

      Description

      When you click on a Job link in the JHS Web UI, it loads the .jhist file. For jobs which have a large number of tasks, the load time can break UI responsiveness.

      1. MAPREDUCE-6376.001.patch
        10 kB
        Ray Chiang
      2. MAPREDUCE-6376.002.patch
        9 kB
        Ray Chiang
      3. MAPREDUCE-6376.003.patch
        10 kB
        Ray Chiang
      4. MAPREDUCE-6376.004.patch
        9 kB
        Ray Chiang
      5. MAPREDUCE-6376.005.patch
        10 kB
        Ray Chiang
      6. MAPREDUCE-6376.006.patch
        10 kB
        Ray Chiang

        Issue Links

          Activity

          Hide
          raviprak Ravi Prakash added a comment -

          Sorry! Never mind. It did go into branch-2.

          Show
          raviprak Ravi Prakash added a comment - Sorry! Never mind. It did go into branch-2.
          Hide
          raviprak Ravi Prakash added a comment -

          Hi Jason! Should this have gone into branch-2 as well?

          Show
          raviprak Ravi Prakash added a comment - Hi Jason! Should this have gone into branch-2 as well?
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2192 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2192/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2192 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2192/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2173 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2173/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2173 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2173/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/234/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #234 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/234/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #244 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/244/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #244 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/244/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #976 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/976/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #976 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/976/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #246 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/246/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #246 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/246/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          rchiang Ray Chiang added a comment -

          Thanks for the keen-eyed reviews and commit!

          Show
          rchiang Ray Chiang added a comment - Thanks for the keen-eyed reviews and commit!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8106 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8106/)
          MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8106 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8106/ ) MAPREDUCE-6376 . Add avro binary support for jhist files. Contributed by Ray Chiang (jlowe: rev 2ac87df578accb6e612f70ded76271cb5082ee10) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestEvents.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventWriter.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          jlowe Jason Lowe added a comment -

          Thanks, Ray! I committed this to trunk and branch-2.

          Show
          jlowe Jason Lowe added a comment - Thanks, Ray! I committed this to trunk and branch-2.
          Hide
          jlowe Jason Lowe added a comment -

          +1. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1. Committing this.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 55s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 javadoc 9m 37s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 49s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 7s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.
              54m 47s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742930/MAPREDUCE-6376.006.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 147e020
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 55s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 javadoc 9m 37s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 49s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 42s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 7s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 45s Tests passed in hadoop-mapreduce-client-core.     54m 47s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742930/MAPREDUCE-6376.006.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 147e020 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5865/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix log file message
          Show
          rchiang Ray Chiang added a comment - Fix log file message
          Hide
          rchiang Ray Chiang added a comment -

          Thanks for catching that.

          I'll update the log message and reserve the fancier conversion for another time.

          Show
          rchiang Ray Chiang added a comment - Thanks for catching that. I'll update the log message and reserve the fancier conversion for another time.
          Hide
          jlowe Jason Lowe added a comment -

          Couple of issues with the new log warning added to JobHistoryEventHandler:

          • JHAdminConfig.DEFAULT_MR_HS_JHIST_FORMAT is not a property but rather a value, so the message is misleading at best. Not sure we need to bother mentioning the default could be wrong, it will just confuse users.
          • Nit: Rather than assume DEFAULT_MR_HS_JHIST_FORMAT is 'json', it would be better to reuse the value, e.g.: "Falling back to default value: " + JHAdminConfig.DEFAULT_MR_HS_JHIST_FORMAT

          Another approach we can take is to take the property value, upper case it, and try to call WriteMode.valueof on it. If we get IllegalArgumentException out of it then we can warn about the fallback to the default value and try again (and let it blow up if it fails, since the default value should not fail). This has the advantage of not being a conversion we manually need to keep in sync with the enum. This isn't a change that has to be made for this JIRA rather just a suggestion.

          Show
          jlowe Jason Lowe added a comment - Couple of issues with the new log warning added to JobHistoryEventHandler: JHAdminConfig.DEFAULT_MR_HS_JHIST_FORMAT is not a property but rather a value, so the message is misleading at best. Not sure we need to bother mentioning the default could be wrong, it will just confuse users. Nit: Rather than assume DEFAULT_MR_HS_JHIST_FORMAT is 'json', it would be better to reuse the value, e.g.: "Falling back to default value: " + JHAdminConfig.DEFAULT_MR_HS_JHIST_FORMAT Another approach we can take is to take the property value, upper case it, and try to call WriteMode.valueof on it. If we get IllegalArgumentException out of it then we can warn about the fallback to the default value and try again (and let it blow up if it fails, since the default value should not fail). This has the advantage of not being a conversion we manually need to keep in sync with the enum. This isn't a change that has to be made for this JIRA rather just a suggestion.
          Hide
          rchiang Ray Chiang added a comment -

          Still seeing the same Findbugs pre-patch as before. Filed MAPREDUCE-6421.

          Show
          rchiang Ray Chiang added a comment - Still seeing the same Findbugs pre-patch as before. Filed MAPREDUCE-6421 .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 0s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 37s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 57s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 8m 54s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core.
              54m 45s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742610/MAPREDUCE-6376.005.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8e33372
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 0s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 57s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 8m 54s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 45s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core.     54m 45s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742610/MAPREDUCE-6376.005.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8e33372 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5858/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Submit correct patch
          • Add else clause after parsing Configuration for jhist property
          Show
          rchiang Ray Chiang added a comment - Submit correct patch Add else clause after parsing Configuration for jhist property
          Hide
          rchiang Ray Chiang added a comment -
          • Add else clause for parsing jhist Configuration property
          Show
          rchiang Ray Chiang added a comment - Add else clause for parsing jhist Configuration property
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch, Ray. Patch looks good overall except for one issue. In JobHistoryEventHandler, we can silently ignore an unrecognized encoding directive and fall back to JSON encoding. This should either throw an error or minimally log a warning stating it doesn't recognize the value, which values are valid, and that it is falling back to json format.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch, Ray. Patch looks good overall except for one issue. In JobHistoryEventHandler, we can silently ignore an unrecognized encoding directive and fall back to JSON encoding. This should either throw an error or minimally log a warning stating it doesn't recognize the value, which values are valid, and that it is falling back to json format.
          Hide
          rchiang Ray Chiang added a comment -

          Ah, I see. Thanks.

          Show
          rchiang Ray Chiang added a comment - Ah, I see. Thanks.
          Hide
          jlowe Jason Lowe added a comment -

          It's complaining about pre-patch trunk, so it has nothing to do with the patch. Why that's a -1 towards the patch itself – I don't know.

          Hoping to get time to look at the updated patch either later today or early next week.

          Show
          jlowe Jason Lowe added a comment - It's complaining about pre-patch trunk, so it has nothing to do with the patch. Why that's a -1 towards the patch itself – I don't know. Hoping to get time to look at the updated patch either later today or early next week.
          Hide
          rchiang Ray Chiang added a comment -

          RE: findbugs

          Not sure how I'm getting an extra findbugs warning out of a file that's not modified by this patch. Does anyone have any suggestions?

          Show
          rchiang Ray Chiang added a comment - RE: findbugs Not sure how I'm getting an extra findbugs warning out of a file that's not modified by this patch. Does anyone have any suggestions?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 47s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 4s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 44s Tests passed in hadoop-mapreduce-client-core.
              55m 23s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12742208/MAPREDUCE-6376.004.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / aa07dea
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 21s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 47s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 43s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 4s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 44s Tests passed in hadoop-mapreduce-client-core.     55m 23s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12742208/MAPREDUCE-6376.004.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / aa07dea Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5845/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Remove else statements from json/binary comparison logic
          • Clean up json/binary logic a bit based on review
          Show
          rchiang Ray Chiang added a comment - Remove else statements from json/binary comparison logic Clean up json/binary logic a bit based on review
          Hide
          rchiang Ray Chiang added a comment -

          Just a quick follow up. I wrote up a quick program to convert old .jhist files into the new binary form and loaded them into a new VM. The speedups are decent at a high # of tasks, but closer to 2x (versus the 3x at <50k tasks).

          751k Mappers/10 Reducers (json)

          • Read Time: 163.8 seconds
          • Event Time: 16.6 seconds

          751k Mappers/10 Reducers (binary)

          • Read Time: 86.7 seconds
          • Event Time: 17.4 seconds

          404k Mappers/4 Reducers (json)

          • Read Time: 93 seconds
          • Event Time: 8.9 seconds

          404k Mappers/4 Reducers (binary)

          • Read Time: 49.1 seconds
          • Event Time: 9.2 seconds

          I'm probably not going to get to a random event generator this week, but this gives us at least two more data points.

          Show
          rchiang Ray Chiang added a comment - Just a quick follow up. I wrote up a quick program to convert old .jhist files into the new binary form and loaded them into a new VM. The speedups are decent at a high # of tasks, but closer to 2x (versus the 3x at <50k tasks). 751k Mappers/10 Reducers (json) Read Time: 163.8 seconds Event Time: 16.6 seconds 751k Mappers/10 Reducers (binary) Read Time: 86.7 seconds Event Time: 17.4 seconds 404k Mappers/4 Reducers (json) Read Time: 93 seconds Event Time: 8.9 seconds 404k Mappers/4 Reducers (binary) Read Time: 49.1 seconds Event Time: 9.2 seconds I'm probably not going to get to a random event generator this week, but this gives us at least two more data points.
          Hide
          rchiang Ray Chiang added a comment -

          Thanks for the feedback. It all sounds good to me.

          IIRC, the else statements were something that helped me get around some unit test failures. I'll have to look into those.

          Show
          rchiang Ray Chiang added a comment - Thanks for the feedback. It all sounds good to me. IIRC, the else statements were something that helped me get around some unit test failures. I'll have to look into those.
          Hide
          jlowe Jason Lowe added a comment -

          Sorry for the delay getting to this.

          I couldn't get access to a machine to generate more than 50k mapper jobs

          It shouldn't be difficult to fake a .jhist file for such a scale with a single machine. Just instantiate an EventWriter and send it TaskStarted/TaskAttemptStarted/MapAttemptFinished/ReduceAttemptFinished/TaskFinished events for as many tasks as desired. Could use a "real" jhist file as a template for the fake events so it has similar sizing for hostnames, counters, etc. Would be good to get some real performance numbers for sizeable jobs rather than guesses as to what it will do in practice.

          Comments on the latest patch:

          +    if (jhistFormat.equals("json")) {
          +      jhistMode = EventWriter.WriteMode.JSON;
          +    } else if (jhistFormat.equals("binary")) {
          +      jhistMode = EventWriter.WriteMode.BINARY;
          +    } else {
          +      jhistMode = EventWriter.WriteMode.JSON;
          +    }
          

          This can be simplified to:

            jhistMode = jhistFormat.equals("binary") ? EventWriter.WriteMode.BINARY : EventWriter.WriteMode.JSON;
          

          but arguably it shouldn't. I'm not a fan of assuming we know the user's intentions when they explicitly overrode the default value to something we don't recognize. IMO if they bothered to override the default to a value we don't recognize then we should throw an error. Case in point: we add a third format later. A user applies that new config with old code. It will silently try to parse json and blow up in cryptic ways rather than tell them in a straightforward error message that we don't know how to do what they're asking and what values we do recognize.

          EventReader should not assume json if it doesn't recognize the version after parsing the schema. We just verified it can only be json or binary, so something's wrong if we get to the else. We should throw here or just assume if it isn't one it must be the other since we just verified it a few lines above. For the latter approach, bonus points for caching the comparison result in a boolean so we don't have to do it all over again.

          Similar situation for the EventWriter constructor and EventWriter.write method. If we don't recognize the WriteMode value then something is very, very wrong. I think we can just validate the WriteMode value in the constructor and then simplify all the other checks by just checking for one and assuming the other, since we already validated in the constructor there's only two possible inputs.

          The writeMode member should be final, as I can't think of any valid reason to allow changing it after the writer is constructed.

          The following code:

              if (!EventWriter.VERSION.equals(version) &&
                  !EventWriter.VERSION_BINARY.equals(version)) {
                 throw new IOException("Incompatible event log version: "+version);
              }
              [...]
              if (EventWriter.VERSION.equals(version)) {
                this.decoder = DecoderFactory.get().jsonDecoder(schema, in);
              } else if (EventWriter.VERSION_BINARY.equals(version)) {
                this.decoder = DecoderFactory.get().binaryDecoder(in, null);
              } else {
                this.decoder = DecoderFactory.get().jsonDecoder(schema, in);
              }
          

          should be simplified to:

              if (EventWriter.VERSION.equals(version)) {
                this.decoder = DecoderFactory.get().jsonDecoder(schema, in);
              } else if (EventWriter.VERSION_BINARY.equals(version)) {
                this.decoder = DecoderFactory.get().binaryDecoder(in, null);
              } else {
                 throw new IOException("Incompatible event log version: "+version);
              }
          

          Nit: "File format generated by the .jhist file generated by the AM" has a bit too much generating. Maybe "File format the AM will use when generating the .jhist file."?

          Show
          jlowe Jason Lowe added a comment - Sorry for the delay getting to this. I couldn't get access to a machine to generate more than 50k mapper jobs It shouldn't be difficult to fake a .jhist file for such a scale with a single machine. Just instantiate an EventWriter and send it TaskStarted/TaskAttemptStarted/MapAttemptFinished/ReduceAttemptFinished/TaskFinished events for as many tasks as desired. Could use a "real" jhist file as a template for the fake events so it has similar sizing for hostnames, counters, etc. Would be good to get some real performance numbers for sizeable jobs rather than guesses as to what it will do in practice. Comments on the latest patch: + if (jhistFormat.equals( "json" )) { + jhistMode = EventWriter.WriteMode.JSON; + } else if (jhistFormat.equals( "binary" )) { + jhistMode = EventWriter.WriteMode.BINARY; + } else { + jhistMode = EventWriter.WriteMode.JSON; + } This can be simplified to: jhistMode = jhistFormat.equals( "binary" ) ? EventWriter.WriteMode.BINARY : EventWriter.WriteMode.JSON; but arguably it shouldn't. I'm not a fan of assuming we know the user's intentions when they explicitly overrode the default value to something we don't recognize. IMO if they bothered to override the default to a value we don't recognize then we should throw an error. Case in point: we add a third format later. A user applies that new config with old code. It will silently try to parse json and blow up in cryptic ways rather than tell them in a straightforward error message that we don't know how to do what they're asking and what values we do recognize. EventReader should not assume json if it doesn't recognize the version after parsing the schema. We just verified it can only be json or binary, so something's wrong if we get to the else. We should throw here or just assume if it isn't one it must be the other since we just verified it a few lines above. For the latter approach, bonus points for caching the comparison result in a boolean so we don't have to do it all over again. Similar situation for the EventWriter constructor and EventWriter.write method. If we don't recognize the WriteMode value then something is very, very wrong. I think we can just validate the WriteMode value in the constructor and then simplify all the other checks by just checking for one and assuming the other, since we already validated in the constructor there's only two possible inputs. The writeMode member should be final, as I can't think of any valid reason to allow changing it after the writer is constructed. The following code: if (!EventWriter.VERSION.equals(version) && !EventWriter.VERSION_BINARY.equals(version)) { throw new IOException( "Incompatible event log version: " +version); } [...] if (EventWriter.VERSION.equals(version)) { this .decoder = DecoderFactory.get().jsonDecoder(schema, in); } else if (EventWriter.VERSION_BINARY.equals(version)) { this .decoder = DecoderFactory.get().binaryDecoder(in, null ); } else { this .decoder = DecoderFactory.get().jsonDecoder(schema, in); } should be simplified to: if (EventWriter.VERSION.equals(version)) { this .decoder = DecoderFactory.get().jsonDecoder(schema, in); } else if (EventWriter.VERSION_BINARY.equals(version)) { this .decoder = DecoderFactory.get().binaryDecoder(in, null ); } else { throw new IOException( "Incompatible event log version: " +version); } Nit: "File format generated by the .jhist file generated by the AM" has a bit too much generating. Maybe "File format the AM will use when generating the .jhist file."?
          Hide
          rchiang Ray Chiang added a comment -

          Jason Lowe, this JHS speedup should have the least side effects (unlike MAPREDUCE-6222 and MAPREDUCE-6394). Let me know what you think when you have time. Thanks.

          Show
          rchiang Ray Chiang added a comment - Jason Lowe , this JHS speedup should have the least side effects (unlike MAPREDUCE-6222 and MAPREDUCE-6394 ). Let me know what you think when you have time. Thanks.
          Hide
          rchiang Ray Chiang added a comment -

          Linking to another JHS speedup JIRA.

          Show
          rchiang Ray Chiang added a comment - Linking to another JHS speedup JIRA.
          Hide
          rchiang Ray Chiang added a comment -

          RE: checkstyle

          Contains an empty if statement, but modifying it would make it uglier for the final "else" case. If someone has a nice suggestion, let me know.

          Show
          rchiang Ray Chiang added a comment - RE: checkstyle Contains an empty if statement, but modifying it would make it uglier for the final "else" case. If someone has a nice suggestion, let me know.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 18m 3s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 38s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 10, now 11).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 9m 31s Tests passed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.
              55m 13s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738991/MAPREDUCE-6376.003.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 95c73d4
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 18m 3s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 38s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 54s The applied patch generated 1 new checkstyle issues (total was 10, now 11). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 3m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 9m 31s Tests passed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 37s Tests passed in hadoop-mapreduce-client-core.     55m 13s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738991/MAPREDUCE-6376.003.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 95c73d4 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5796/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix initializing corner cases. Should fix unit tests issues.
          Show
          rchiang Ray Chiang added a comment - Fix initializing corner cases. Should fix unit tests issues.
          Hide
          rchiang Ray Chiang added a comment -

          Ugh, put up wrong patch.

          Show
          rchiang Ray Chiang added a comment - Ugh, put up wrong patch.
          Hide
          rchiang Ray Chiang added a comment -

          Ugh, put up wrong patch.

          Show
          rchiang Ray Chiang added a comment - Ugh, put up wrong patch.
          Hide
          rchiang Ray Chiang added a comment -
          • Fix one area of propagation for skipCounters
          Show
          rchiang Ray Chiang added a comment - Fix one area of propagation for skipCounters
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 19m 8s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 54s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 3s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 35s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 9m 0s Tests failed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 42s Tests passed in hadoop-mapreduce-client-core.
              56m 34s  



          Reason Tests
          Failed unit tests hadoop.mapreduce.v2.app.TestRecovery
            hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
            hadoop.mapreduce.v2.app.TestMRApp
            hadoop.mapreduce.v2.app.TestFetchFailure
            hadoop.mapreduce.v2.app.TestStagingCleanup
            hadoop.mapreduce.v2.app.TestAMInfos
            hadoop.mapreduce.v2.app.TestMRAppComponentDependencies
            hadoop.mapreduce.jobhistory.TestEvents
            hadoop.mapreduce.v2.app.TestMRAppMaster



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738929/MAPREDUCE-6376.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 19m 8s Pre-patch trunk has 1 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 54s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 3s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 44s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 9m 0s Tests failed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 47s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 42s Tests passed in hadoop-mapreduce-client-core.     56m 34s   Reason Tests Failed unit tests hadoop.mapreduce.v2.app.TestRecovery   hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler   hadoop.mapreduce.v2.app.TestMRApp   hadoop.mapreduce.v2.app.TestFetchFailure   hadoop.mapreduce.v2.app.TestStagingCleanup   hadoop.mapreduce.v2.app.TestAMInfos   hadoop.mapreduce.v2.app.TestMRAppComponentDependencies   hadoop.mapreduce.jobhistory.TestEvents   hadoop.mapreduce.v2.app.TestMRAppMaster Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738929/MAPREDUCE-6376.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5793/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          RE: findbugs

          Getting a lot of these empty html messages. Filed HADOOP-12083 for investigation.

          RE: failed unit tests

          All failed tests pass in my tree.

          Show
          rchiang Ray Chiang added a comment - RE: findbugs Getting a lot of these empty html messages. Filed HADOOP-12083 for investigation. RE: failed unit tests All failed tests pass in my tree.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 35s Findbugs (version 3.0.0) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 9m 44s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 42s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          -1 findbugs 3m 48s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 9m 20s Tests failed in hadoop-mapreduce-client-app.
          +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common.
          +1 mapreduce tests 1m 40s Tests passed in hadoop-mapreduce-client-core.
              55m 0s  



          Reason Tests
          FindBugs module:hadoop-mapreduce-client-app
          Failed unit tests hadoop.mapreduce.v2.app.TestRecovery
            hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler
            hadoop.mapreduce.v2.app.TestMRApp
            hadoop.mapreduce.v2.app.TestFetchFailure
            hadoop.mapreduce.v2.app.TestStagingCleanup
            hadoop.mapreduce.v2.app.TestAMInfos
            hadoop.mapreduce.v2.app.TestMRAppComponentDependencies
            hadoop.mapreduce.jobhistory.TestEvents
            hadoop.mapreduce.v2.app.TestMRAppMaster



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738929/MAPREDUCE-6376.002.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
          hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 35s Findbugs (version 3.0.0) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 9m 44s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 42s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 findbugs 3m 48s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 9m 20s Tests failed in hadoop-mapreduce-client-app. +1 mapreduce tests 0m 46s Tests passed in hadoop-mapreduce-client-common. +1 mapreduce tests 1m 40s Tests passed in hadoop-mapreduce-client-core.     55m 0s   Reason Tests FindBugs module:hadoop-mapreduce-client-app Failed unit tests hadoop.mapreduce.v2.app.TestRecovery   hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler   hadoop.mapreduce.v2.app.TestMRApp   hadoop.mapreduce.v2.app.TestFetchFailure   hadoop.mapreduce.v2.app.TestStagingCleanup   hadoop.mapreduce.v2.app.TestAMInfos   hadoop.mapreduce.v2.app.TestMRAppComponentDependencies   hadoop.mapreduce.jobhistory.TestEvents   hadoop.mapreduce.v2.app.TestMRAppMaster Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738929/MAPREDUCE-6376.002.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 Findbugs warnings https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html hadoop-mapreduce-client-app test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt hadoop-mapreduce-client-common test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5791/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Update to apply cleanly after MAPREDUCE-6350.

          Show
          rchiang Ray Chiang added a comment - Update to apply cleanly after MAPREDUCE-6350 .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12738679/MAPREDUCE-6376.001.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a7a7768
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5790/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12738679/MAPREDUCE-6376.001.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a7a7768 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5790/console This message was automatically generated.
          Hide
          rchiang Ray Chiang added a comment -

          Submit patch for testing

          Show
          rchiang Ray Chiang added a comment - Submit patch for testing
          Hide
          rchiang Ray Chiang added a comment -

          A few comments:

          1) It turns out that Avro parsing is anywhere from 70% to 90% of the .jhist processing time. Some data points for the json .jhist file:

          • 50k mappers
            • 20 seconds overall read time
            • 16.6 seconds Avro parsing/reading
          • 404k mappers
            • 68 seconds
            • 49 seconds Avro parsing/reading
            • 751k mappers
            • 300 seconds
            • 280 seconds Avro parsing/reading

          2) I couldn't get access to a machine to generate more than 50k mapper jobs, but my rough experiments would see about 4x to 5x speedup in Avro parsing/reading. For the worst case improvement on 751k mappers, I would expect the 300 seconds of processing time to get down to about 90 seconds. There is room to shave down the processing time by a few seconds here and there, but that's probably better left to subsequent JIRAs.

          3) The .jhist file output format is now a configuration option, with the default set to json.

          Show
          rchiang Ray Chiang added a comment - A few comments: 1) It turns out that Avro parsing is anywhere from 70% to 90% of the .jhist processing time. Some data points for the json .jhist file: 50k mappers 20 seconds overall read time 16.6 seconds Avro parsing/reading 404k mappers 68 seconds 49 seconds Avro parsing/reading 751k mappers 300 seconds 280 seconds Avro parsing/reading 2) I couldn't get access to a machine to generate more than 50k mapper jobs, but my rough experiments would see about 4x to 5x speedup in Avro parsing/reading. For the worst case improvement on 751k mappers, I would expect the 300 seconds of processing time to get down to about 90 seconds. There is room to shave down the processing time by a few seconds here and there, but that's probably better left to subsequent JIRAs. 3) The .jhist file output format is now a configuration option, with the default set to json.
          Hide
          rchiang Ray Chiang added a comment -
          • Add option to write .jhist files using Avro binaryEncoder
          • Modify jhist file header to determine which Avro decoder to use to read the rest of the .jhist file
          Show
          rchiang Ray Chiang added a comment - Add option to write .jhist files using Avro binaryEncoder Modify jhist file header to determine which Avro decoder to use to read the rest of the .jhist file
          Hide
          rchiang Ray Chiang added a comment -

          Original JIRA.

          Show
          rchiang Ray Chiang added a comment - Original JIRA.

            People

            • Assignee:
              rchiang Ray Chiang
              Reporter:
              rchiang Ray Chiang
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development