Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We need to be able to understand the capacity model for the timeline server to give users the tools they need to deploy a timeline server with the correct capacity.

      I propose we create a mapreduce job that can measure timeline server write and read performance. Transactions per second, I/O for both read and write would be a good start.

      This could be done as an example or test job that could be tied into gridmix.

      1. YARN-2556.15.patch
        43 kB
        Chang Li
      2. YARN-2556.14.whitespacefix.patch
        43 kB
        Chang Li
      3. YARN-2556.14.patch
        39 kB
        Chang Li
      4. YARN-2556.13.whitespacefix.patch
        43 kB
        Chang Li
      5. YARN-2556.13.patch
        39 kB
        Chang Li
      6. YARN-2556.12.patch
        39 kB
        Chang Li
      7. YARN-2556.11.patch
        37 kB
        Chang Li
      8. YARN-2556.10.patch
        37 kB
        Chang Li
      9. YARN-2556.9.patch
        16 kB
        Chang Li
      10. YARN-2556.8.patch
        19 kB
        Chang Li
      11. YARN-2556.7.patch
        19 kB
        Chang Li
      12. YARN-2556.6.patch
        5 kB
        Chang Li
      13. YARN-2556.5.patch
        19 kB
        Chang Li
      14. YARN-2556.4.patch
        19 kB
        Chang Li
      15. YARN-2556.3.patch
        15 kB
        Chang Li
      16. YARN-2556.2.patch
        11 kB
        Jonathan Eagles
      17. YARN-2556.1.patch
        12 kB
        Jonathan Eagles
      18. YARN-2556.patch
        12 kB
        Amit Tiwari
      19. yarn2556.patch
        10 kB
        Chang Li
      20. yarn2556.patch
        10 kB
        Chang Li
      21. YARN-2556-WIP.patch
        8 kB
        Chang Li
      22. yarn2556_wip.patch
        9 kB
        Chang Li
      23. YARN-2556-WIP.patch
        8 kB
        Chen He

        Issue Links

          Activity

          Hide
          jeagles Jonathan Eagles added a comment -

          This jira is to give users realistic performance numbers running the timeline server for their setup specific to their own hardware. (HBase, leveldb, etc)

          FYI. LevelDB publishes their own performance statics.
          https://code.google.com/p/leveldb/

          Show
          jeagles Jonathan Eagles added a comment - This jira is to give users realistic performance numbers running the timeline server for their setup specific to their own hardware. (HBase, leveldb, etc) FYI. LevelDB publishes their own performance statics. https://code.google.com/p/leveldb/
          Hide
          zjshen Zhijie Shen added a comment -

          Jonathan Eagles, it sounds an interesting work. Is it possible to see the throughput difference between TimelineDataManager and the web front interface? I suspect the web front interface is going to be bottleneck to throttle the end-to-end performance. With this analysis, we can have clearer picture about the reasonable to timeline server instances required to get rid of web font interface bottleneck (YARN-2520).

          Show
          zjshen Zhijie Shen added a comment - Jonathan Eagles , it sounds an interesting work. Is it possible to see the throughput difference between TimelineDataManager and the web front interface? I suspect the web front interface is going to be bottleneck to throttle the end-to-end performance. With this analysis, we can have clearer picture about the reasonable to timeline server instances required to get rid of web font interface bottleneck ( YARN-2520 ).
          Hide
          airbots Chen He added a comment -

          A working in progress patch. Any reply will be appreciated.

          Show
          airbots Chen He added a comment - A working in progress patch. Any reply will be appreciated.
          Hide
          lichangleo Chang Li added a comment -

          Thanks Chen He for the substantial early work! I have moved the test job into mapreduce jobclient tests to avoid circular dependency. I have tested the patch, and it has successfully shown the write time, write counters and write per second. I will continue to work on it to add more metric of measurement such as transaction rates, IO rates and memory usage.

          Show
          lichangleo Chang Li added a comment - Thanks Chen He for the substantial early work! I have moved the test job into mapreduce jobclient tests to avoid circular dependency. I have tested the patch, and it has successfully shown the write time, write counters and write per second. I will continue to work on it to add more metric of measurement such as transaction rates, IO rates and memory usage.
          Hide
          lichangleo Chang Li added a comment -

          current work in progress patch. implement the measure for iorate and transaction rate

          Show
          lichangleo Chang Li added a comment - current work in progress patch. implement the measure for iorate and transaction rate
          Hide
          lichangleo Chang Li added a comment -

          Cleaned up my patch, welcome to review. I have used this application to test the timeline server throughput on local mode by launching 4 mappers and each will put an entity larger than 100 kbs and iterate for 1000 times. Here is my measure result, on my local machine, the timeline server can provide about 10Mbs io rate for write. There is some deviation from the write throughput for leveldb. People are welcome to try this tool and comment about it.

          Show
          lichangleo Chang Li added a comment - Cleaned up my patch, welcome to review. I have used this application to test the timeline server throughput on local mode by launching 4 mappers and each will put an entity larger than 100 kbs and iterate for 1000 times. Here is my measure result, on my local machine, the timeline server can provide about 10Mbs io rate for write. There is some deviation from the write throughput for leveldb. People are welcome to try this tool and comment about it.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12678020/yarn2556.patch
          against trunk revision d33e07d.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5631//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12678020/yarn2556.patch against trunk revision d33e07d. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5631//console This message is automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          attempted modified patch according to build failure

          Show
          lichangleo Chang Li added a comment - attempted modified patch according to build failure
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12681427/yarn2556.patch
          against trunk revision 1a2e5cb.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.fs.slive.TestSlive
          org.apache.hadoop.mapred.TestJobSysDirWithDFS

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5840//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5840//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681427/yarn2556.patch against trunk revision 1a2e5cb. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.fs.slive.TestSlive org.apache.hadoop.mapred.TestJobSysDirWithDFS +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5840//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5840//console This message is automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          I have run those failed tests on my local machine and they all passed with my patch

          Show
          lichangleo Chang Li added a comment - I have run those failed tests on my local machine and they all passed with my patch
          Hide
          airbots Chen He added a comment -

          Current benchmark only contains basic Timelineserver write / sec. Do we need to add more?

          Show
          airbots Chen He added a comment - Current benchmark only contains basic Timelineserver write / sec. Do we need to add more?
          Hide
          tiwari Amit Tiwari added a comment -

          Hi guys,
          I've done the following enhancements to the previous patches that were posted:
          1) Earlier, the payload was getting set as the entityId. Since the entityId is used as a key, by LevelDB it was crashing under moderate loads, because each key size was ~2MB. Hence I've changed it to send the payload as a part of OtherInfo. This is handled well.
          2) Instead of posting a string of repeated 'a's as a payload, I choose from a set of characters. This ensures that the LevelDB does not get away easily with compression ('cos algos can easily compress a string if it comprises a single repeated character)

          Here are some of the performance numbers that I've got:
          I run 20 concurrent jobs, with the argument -m 300 -s 10 -t 20
          On a 36 node cluster, this results in ~830 concurrent containers (e.g maps), each firing 10KB of payload, 20 times.

          Level DB seems to hold up fine.

          Would you have other ways that I could stress/load the system even more?
          thanks
          --amit

          Show
          tiwari Amit Tiwari added a comment - Hi guys, I've done the following enhancements to the previous patches that were posted: 1) Earlier, the payload was getting set as the entityId. Since the entityId is used as a key, by LevelDB it was crashing under moderate loads, because each key size was ~2MB. Hence I've changed it to send the payload as a part of OtherInfo. This is handled well. 2) Instead of posting a string of repeated 'a's as a payload, I choose from a set of characters. This ensures that the LevelDB does not get away easily with compression ('cos algos can easily compress a string if it comprises a single repeated character) Here are some of the performance numbers that I've got: I run 20 concurrent jobs, with the argument -m 300 -s 10 -t 20 On a 36 node cluster, this results in ~830 concurrent containers (e.g maps), each firing 10KB of payload, 20 times. Level DB seems to hold up fine. Would you have other ways that I could stress/load the system even more? thanks --amit
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12699764/YARN-2556.patch
          against trunk revision d49ae72.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestJobConf

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6675//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6675//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12699764/YARN-2556.patch against trunk revision d49ae72. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestJobConf Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6675//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6675//console This message is automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Chang Li and Amit Tiwari. Got around to trying this out. Uploading a new patch that fixes some bugs that were present. My main issue is that I wasn't able to push the timeline server hard enough from a small number of nodes. There was also some integer rounding issues and formatting issues.

          Show
          jeagles Jonathan Eagles added a comment - Chang Li and Amit Tiwari . Got around to trying this out. Uploading a new patch that fixes some bugs that were present. My main issue is that I wasn't able to push the timeline server hard enough from a small number of nodes. There was also some integer rounding issues and formatting issues.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12704923/YARN-2556.1.patch
          against trunk revision 2681ed9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6988//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6988//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12704923/YARN-2556.1.patch against trunk revision 2681ed9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/6988//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6988//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12705194/YARN-2556.2.patch
          against trunk revision 968425e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7005//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7005//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12705194/YARN-2556.2.patch against trunk revision 968425e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7005//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7005//console This message is automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I'm looking at this as part of a parallel effort for the timeline service v.2 (YARN-3437). While adopting it for the timeline service v.2, I had some review comments for the latest patch here.

          (1) defaults
          The way defaults work is a little bit counter-intuitive and surprising. It turns out you have to provide most of the parameters even if you're happy with the defaults. For example, you cannot omit "-s". The only way you take advantage of the default is by providing zero or a negative value.

          IMO, it would be great the defaults are assumed unless the user overrode a value specifically. In other words, parameters can be entirely optional.

          (2) printUsage
          There is a line break missing in printUsage (line 51). Also, related to the above, it would be great if we specify the default for each option.

          (3) conf option
          Just noting that the conf option is not really consumed by this app, because it's consumed by the GenericOptionsParser. So I'm not sure why the code that parses the conf option exists here?

          (4) constants and strings
          Nit: it would be good to lift all the constants and repeated strings into static final variables.

          (5) System.currentTimeMillis()
          System.currentTimeMillis() might not be a good choice for the timer. I would suggest using System.nanoTime().

          (6) printing out aggregate metrics
          It prints out transaction rate and the IO rate per mapper, but it would be good to report total transaction rate and IO rate as well.

          Do let me know if you have any questions on this...

          Show
          sjlee0 Sangjin Lee added a comment - I'm looking at this as part of a parallel effort for the timeline service v.2 ( YARN-3437 ). While adopting it for the timeline service v.2, I had some review comments for the latest patch here. (1) defaults The way defaults work is a little bit counter-intuitive and surprising. It turns out you have to provide most of the parameters even if you're happy with the defaults. For example, you cannot omit "-s". The only way you take advantage of the default is by providing zero or a negative value. IMO, it would be great the defaults are assumed unless the user overrode a value specifically. In other words, parameters can be entirely optional. (2) printUsage There is a line break missing in printUsage (line 51). Also, related to the above, it would be great if we specify the default for each option. (3) conf option Just noting that the conf option is not really consumed by this app, because it's consumed by the GenericOptionsParser. So I'm not sure why the code that parses the conf option exists here? (4) constants and strings Nit: it would be good to lift all the constants and repeated strings into static final variables. (5) System.currentTimeMillis() System.currentTimeMillis() might not be a good choice for the timer. I would suggest using System.nanoTime(). (6) printing out aggregate metrics It prints out transaction rate and the IO rate per mapper, but it would be good to report total transaction rate and IO rate as well. Do let me know if you have any questions on this...
          Hide
          zjshen Zhijie Shen added a comment -

          Do we have any plan to get this jira in recently? /cc Jonathan Eagles.

          Show
          zjshen Zhijie Shen added a comment - Do we have any plan to get this jira in recently? /cc Jonathan Eagles .
          Hide
          jeagles Jonathan Eagles added a comment -

          I'm very swamped current. Even though it would take very little time to address this, I just can't find the time. Please let's just move on and I will get to this in time.

          Show
          jeagles Jonathan Eagles added a comment - I'm very swamped current. Even though it would take very little time to address this, I just can't find the time. Please let's just move on and I will get to this in time.
          Hide
          lichangleo Chang Li added a comment -

          Thanks Sangjin Lee for thorough review! have updated patch according to those suggestions. Also enable user to configure how many primary filters and related entities should be added to the testing timeline entities.

          Show
          lichangleo Chang Li added a comment - Thanks Sangjin Lee for thorough review! have updated patch according to those suggestions. Also enable user to configure how many primary filters and related entities should be added to the testing timeline entities.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 14s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 32s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 36s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 105m 46s Tests passed in hadoop-mapreduce-client-jobclient.
              122m 20s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732105/YARN-2556.3.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 3d28611
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7873/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7873/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7873/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7873/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 14s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 36s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 105m 46s Tests passed in hadoop-mapreduce-client-jobclient.     122m 20s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732105/YARN-2556.3.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 3d28611 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7873/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7873/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7873/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7873/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          fix existing whitespace issue in MapredTestDrive

          Show
          lichangleo Chang Li added a comment - fix existing whitespace issue in MapredTestDrive
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732249/YARN-2556.4.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 6d5da94
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7888/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732249/YARN-2556.4.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 6d5da94 Console output https://builds.apache.org/job/PreCommit-YARN-Build/7888/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 20s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 46s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 33s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 mapreduce tests 109m 52s Tests failed in hadoop-mapreduce-client-jobclient.
              126m 40s  



          Reason Tests
          Timed out tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732269/YARN-2556.5.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f4e2b3c
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7891/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7891/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7891/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7891/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 46s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 33s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 mapreduce tests 109m 52s Tests failed in hadoop-mapreduce-client-jobclient.     126m 40s   Reason Tests Timed out tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732269/YARN-2556.5.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f4e2b3c whitespace https://builds.apache.org/job/PreCommit-YARN-Build/7891/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7891/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7891/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7891/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 13s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          -1 javac 4m 43s The patch appears to cause the build to fail.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732342/YARN-2556.6.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f24452d
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7897/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 13s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. -1 javac 4m 43s The patch appears to cause the build to fail. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732342/YARN-2556.6.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f24452d Console output https://builds.apache.org/job/PreCommit-YARN-Build/7897/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 24s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 22s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 mapreduce tests 74m 8s Tests failed in hadoop-mapreduce-client-jobclient.
              90m 38s  



          Reason Tests
          Failed unit tests hadoop.mapreduce.v2.TestSpeculativeExecution
            hadoop.mapreduce.v2.TestMRJobsWithHistoryService
            hadoop.mapreduce.security.TestJHSSecurity
            hadoop.mapreduce.v2.TestNonExistentJob
            hadoop.mapreduce.v2.TestMiniMRProxyUser
            hadoop.mapreduce.lib.output.TestJobOutputCommitter
            hadoop.mapreduce.v2.TestMRAppWithCombiner
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapreduce.v2.TestUberAM
            hadoop.mapreduce.security.TestMRCredentials
            hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities
            hadoop.mapreduce.v2.TestRMNMInfo
            hadoop.mapreduce.v2.TestMRJobs
          Timed out tests org.apache.hadoop.mapreduce.v2.TestMROldApiJobs



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732354/YARN-2556.7.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / f24452d
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7901/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7901/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7901/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 mapreduce tests 74m 8s Tests failed in hadoop-mapreduce-client-jobclient.     90m 38s   Reason Tests Failed unit tests hadoop.mapreduce.v2.TestSpeculativeExecution   hadoop.mapreduce.v2.TestMRJobsWithHistoryService   hadoop.mapreduce.security.TestJHSSecurity   hadoop.mapreduce.v2.TestNonExistentJob   hadoop.mapreduce.v2.TestMiniMRProxyUser   hadoop.mapreduce.lib.output.TestJobOutputCommitter   hadoop.mapreduce.v2.TestMRAppWithCombiner   hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapreduce.v2.TestUberAM   hadoop.mapreduce.security.TestMRCredentials   hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities   hadoop.mapreduce.v2.TestRMNMInfo   hadoop.mapreduce.v2.TestMRJobs Timed out tests org.apache.hadoop.mapreduce.v2.TestMROldApiJobs Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732354/YARN-2556.7.patch Optional Tests javac unit findbugs checkstyle git revision trunk / f24452d hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7901/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7901/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7901/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks for the latest patch Chang Li!

          You might want to take a look at MAPREDUCE-6335 and MAPREDUCE-6337 to see some of the changes and comments made in there. We'll need to consider how we can reconcile these two. It would be GREAT if we have this patch in the form so that the future reconciliation would be very straightforward. The concerns might involve having a place to switch between the timeline service v1 and timeline service v.2, and adopting refactoring that was done in the v.2 test. For example, you'll see that in MAPREDUCE-6337 we separated the actual entity creation logic almost out of the test driver itself. It'd be ideal if the test driver is agnostic of the timeline service v.1 or v.2, and those are deferred to the actual mappers. Again, not everything can be cleanly separated, but to the extent we can, it would be great. Thanks!

          By the way, this might need to be moved to the MAPREDUCE project as this touches the mapreduce code although it is supposed to test the timeline service.

          Some more specific comments:
          (TimelineServerPerformance.java)

          • l.45: unused incorrect import
          • l.54-57: constants should follow the naming convention (upper case)
          • l.64: it was pointed out on the v.2 test, but using "Tez" here is rather awkward as this is in the mapreduce code; can we use a generic name?
          • l.89: nit: indentation
          • l.245: why is System.runFinalization() necessary? I don't see a compelling reason why it should be called here (it was also pointed out in the other JIRA)
          • l.254: bug: i++ => t++

          (MapredTestDriver.java)

          • it looks like there are a lot of whitespace changes here; let's not introduce more changes...
          Show
          sjlee0 Sangjin Lee added a comment - Thanks for the latest patch Chang Li ! You might want to take a look at MAPREDUCE-6335 and MAPREDUCE-6337 to see some of the changes and comments made in there. We'll need to consider how we can reconcile these two. It would be GREAT if we have this patch in the form so that the future reconciliation would be very straightforward. The concerns might involve having a place to switch between the timeline service v1 and timeline service v.2, and adopting refactoring that was done in the v.2 test. For example, you'll see that in MAPREDUCE-6337 we separated the actual entity creation logic almost out of the test driver itself. It'd be ideal if the test driver is agnostic of the timeline service v.1 or v.2, and those are deferred to the actual mappers. Again, not everything can be cleanly separated, but to the extent we can, it would be great. Thanks! By the way, this might need to be moved to the MAPREDUCE project as this touches the mapreduce code although it is supposed to test the timeline service. Some more specific comments: (TimelineServerPerformance.java) l.45: unused incorrect import l.54-57: constants should follow the naming convention (upper case) l.64: it was pointed out on the v.2 test, but using "Tez" here is rather awkward as this is in the mapreduce code; can we use a generic name? l.89: nit: indentation l.245: why is System.runFinalization() necessary? I don't see a compelling reason why it should be called here (it was also pointed out in the other JIRA) l.254: bug: i++ => t++ (MapredTestDriver.java) it looks like there are a lot of whitespace changes here; let's not introduce more changes...
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 16s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 35s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 31s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 102m 55s Tests passed in hadoop-mapreduce-client-jobclient.
              119m 26s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12732945/YARN-2556.8.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 15ccd96
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7941/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7941/testReport/
          Java 1.7.0_55
          uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/7941/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 16s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 35s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 31s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 40s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 102m 55s Tests passed in hadoop-mapreduce-client-jobclient.     119m 26s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12732945/YARN-2556.8.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 15ccd96 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/7941/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7941/testReport/ Java 1.7.0_55 uname Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/7941/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee thanks a lot for review and pointing me to those two very helpful jiras! I have updated my patch by following the style you did in TimelineServicePerformanceV2, and refactor the entities creation and entities put work into a separate SimpleEntityWriterV1 mapper. I have also enabled switch between v1 and v2. But I haven't import the Job History File Replay Mapper yet, do I also need to? Thanks!

          Show
          lichangleo Chang Li added a comment - Sangjin Lee thanks a lot for review and pointing me to those two very helpful jiras! I have updated my patch by following the style you did in TimelineServicePerformanceV2, and refactor the entities creation and entities put work into a separate SimpleEntityWriterV1 mapper. I have also enabled switch between v1 and v2. But I haven't import the Job History File Replay Mapper yet, do I also need to? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 33s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 102m 43s Tests failed in hadoop-mapreduce-client-jobclient.
              119m 4s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestMRTimelineEventHandling



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12733931/YARN-2556.9.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / e422e76
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8008/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8008/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8008/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8008/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 33s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 41s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 102m 43s Tests failed in hadoop-mapreduce-client-jobclient.     119m 4s   Reason Tests Failed unit tests hadoop.mapred.TestMRTimelineEventHandling Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12733931/YARN-2556.9.patch Optional Tests javac unit findbugs checkstyle git revision trunk / e422e76 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8008/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8008/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8008/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8008/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Add JobHistoryFileReplayMapper mapper

          Show
          lichangleo Chang Li added a comment - Add JobHistoryFileReplayMapper mapper
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 53s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 9m 47s There were no new javac warning messages.
          +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 19s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 2m 2s mvn install still works.
          +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse.
          +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 99m 47s Tests failed in hadoop-mapreduce-client-jobclient.
              120m 54s  



          Reason Tests
          Timed out tests org.apache.hadoop.mapred.TestMerge



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12734234/YARN-2556.10.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 03f897f
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8031/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8031/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8031/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8031/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 53s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 9m 47s There were no new javac warning messages. +1 release audit 0m 31s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 19s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 2m 2s mvn install still works. +1 eclipse:eclipse 0m 39s The patch built with eclipse:eclipse. +1 findbugs 0m 51s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 99m 47s Tests failed in hadoop-mapreduce-client-jobclient.     120m 54s   Reason Tests Timed out tests org.apache.hadoop.mapred.TestMerge Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12734234/YARN-2556.10.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 03f897f whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8031/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8031/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8031/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8031/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Have test TestMerge on my on machine, it passed with my patch on. It's not caused by my change. That whitespace issue is also introduced by existing code. Sangjin Lee, Junping Du, Zhijie Shen could you please help review the latest patch. Thanks!

          Show
          lichangleo Chang Li added a comment - Have test TestMerge on my on machine, it passed with my patch on. It's not caused by my change. That whitespace issue is also introduced by existing code. Sangjin Lee , Junping Du , Zhijie Shen could you please help review the latest patch. Thanks!
          Hide
          sjlee0 Sangjin Lee added a comment -

          Chang Li, thanks much for updating the patch! I'm sorry I didn't get to review this sooner. I'll spend some time today to review this in more detail.

          That said, I'd like to raise one important point as a goal: it would be great if we can have a single TimelineServicePerformance class that can accommodate both v1 and v2 timeline service tests. That way, any v1- or v2-specific code can be isolated to the specific writers. If we came this close but are still left with two versions of TimelineServicePerformance, that would not be such a great outcome.

          I think the latest patch is almost there. We might need to make the writer base class more generic, as it still involves types that are specific to v1. Perhaps we can simply use the mapper class in TimelineServicePerformance (i.e. remove EntityWriter altogether from TimelineServicePerformance), and move any v1- or v2-specific writers into their own classes outside TimelineServicePerformance.

          I know the option handling is going to be a sticky point, as it will be difficult to have specific mappers inject their own option handling. Maybe that's something we might have to live with. I'm curious if you have ideas to make even that generic.

          Show
          sjlee0 Sangjin Lee added a comment - Chang Li , thanks much for updating the patch! I'm sorry I didn't get to review this sooner. I'll spend some time today to review this in more detail. That said, I'd like to raise one important point as a goal: it would be great if we can have a single TimelineServicePerformance class that can accommodate both v1 and v2 timeline service tests . That way, any v1- or v2-specific code can be isolated to the specific writers. If we came this close but are still left with two versions of TimelineServicePerformance, that would not be such a great outcome. I think the latest patch is almost there. We might need to make the writer base class more generic, as it still involves types that are specific to v1. Perhaps we can simply use the mapper class in TimelineServicePerformance (i.e. remove EntityWriter altogether from TimelineServicePerformance), and move any v1- or v2-specific writers into their own classes outside TimelineServicePerformance. I know the option handling is going to be a sticky point, as it will be difficult to have specific mappers inject their own option handling. Maybe that's something we might have to live with. I'm curious if you have ideas to make even that generic.
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee Thanks so much for review and providing valuable suggestions! I have removed entityWriter from TimelineServicePerformance. For now we are going to have separate simpleEntityWriter and JobHistoryFileReplayMapper mappers files/classes for V1 and V2. Let me know what else I can try to make it better. Thanks!

          Show
          lichangleo Chang Li added a comment - Sangjin Lee Thanks so much for review and providing valuable suggestions! I have removed entityWriter from TimelineServicePerformance. For now we are going to have separate simpleEntityWriter and JobHistoryFileReplayMapper mappers files/classes for V1 and V2. Let me know what else I can try to make it better. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 58s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 6 new or modified test files.
          +1 javac 8m 47s There were no new javac warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 39s There were no new checkstyle issues.
          -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 42s mvn install still works.
          +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse.
          +1 findbugs 0m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 86m 41s Tests failed in hadoop-mapreduce-client-jobclient.
              106m 46s  



          Reason Tests
          Failed unit tests hadoop.mapred.join.TestDatamerge
            hadoop.mapred.TestLazyOutput
            hadoop.mapred.TestReduceFetch
            hadoop.mapred.TestJobName
          Timed out tests org.apache.hadoop.mapreduce.v2.TestMRJobs
            org.apache.hadoop.mapred.TestMRIntermediateDataEncryption



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12735924/YARN-2556.11.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 246cefa
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8116/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8116/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8116/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8116/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 58s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 6 new or modified test files. +1 javac 8m 47s There were no new javac warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 39s There were no new checkstyle issues. -1 whitespace 0m 1s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 42s mvn install still works. +1 eclipse:eclipse 0m 37s The patch built with eclipse:eclipse. +1 findbugs 0m 55s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 86m 41s Tests failed in hadoop-mapreduce-client-jobclient.     106m 46s   Reason Tests Failed unit tests hadoop.mapred.join.TestDatamerge   hadoop.mapred.TestLazyOutput   hadoop.mapred.TestReduceFetch   hadoop.mapred.TestJobName Timed out tests org.apache.hadoop.mapreduce.v2.TestMRJobs   org.apache.hadoop.mapred.TestMRIntermediateDataEncryption Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12735924/YARN-2556.11.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 246cefa whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8116/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8116/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8116/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8116/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Chang Li, thanks much for the latest patch! I feel that it's real close. I like that it's much more modular and can support v2 more easily. I do have a few more comments and suggestions.

          (TimelineServicePerformance.java)

          • l.35-36: there should be no timeline type imports here; unused imports?
          • l.201-212: it's an oversight, but this belongs in the SimpleEntityWriter class, and should be moved there

          (JobHistoryFileReplayMapperV1.java)

          • Can you refactor this as much as possible so that v1 and v2 do not duplicate any shared code? For example, JobFiles, constants, and some operations inside map() are clearly common between v1 and v2. It might involve extracting some common (helper) methods.
          • l.162: We found this with the v2 code, but there is a bug here: it is possible that the JobFiles instance may not have both the jobhistory file and the configuration file. We should skip processing the JobFiles instance if either is null.
          Show
          sjlee0 Sangjin Lee added a comment - Chang Li , thanks much for the latest patch! I feel that it's real close. I like that it's much more modular and can support v2 more easily. I do have a few more comments and suggestions. (TimelineServicePerformance.java) l.35-36: there should be no timeline type imports here; unused imports? l.201-212: it's an oversight, but this belongs in the SimpleEntityWriter class, and should be moved there (JobHistoryFileReplayMapperV1.java) Can you refactor this as much as possible so that v1 and v2 do not duplicate any shared code? For example, JobFiles, constants, and some operations inside map() are clearly common between v1 and v2. It might involve extracting some common (helper) methods. l.162: We found this with the v2 code, but there is a bug here: it is possible that the JobFiles instance may not have both the jobhistory file and the configuration file. We should skip processing the JobFiles instance if either is null.
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee Thanks so much for patiently guiding me to improve the work! I have updated my patch according to your suggestions and create a JobHistoryFileReplayHelper to extract those common code between v1 and v2 JobHistoryFileReplay mappers. Please let me know what you think of the latest patch. Thanks!

          Show
          lichangleo Chang Li added a comment - Sangjin Lee Thanks so much for patiently guiding me to improve the work! I have updated my patch according to your suggestions and create a JobHistoryFileReplayHelper to extract those common code between v1 and v2 JobHistoryFileReplay mappers. Please let me know what you think of the latest patch. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 5m 40s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 13s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse.
          +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 103m 53s Tests passed in hadoop-mapreduce-client-jobclient.
              120m 36s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736047/YARN-2556.12.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / d725dd8
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8128/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8128/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8128/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8128/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 5m 40s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 13s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 31s The patch built with eclipse:eclipse. +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 103m 53s Tests passed in hadoop-mapreduce-client-jobclient.     120m 36s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736047/YARN-2556.12.patch Optional Tests javac unit findbugs checkstyle git revision trunk / d725dd8 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8128/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8128/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8128/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8128/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          It looks pretty good! These are mostly minor comments. Hopefully it won't be too hard to address them. Also, do take a look at the whitespace warning to clear up the jenkins.

          (JobHistoryFileReplayHelper.java)

          • l.37: the class doesn't need to be public
          • l.46: JOB_ID_PARSER can be private
          • l.48: FileType can be private
          • l.53: nit: spacing (no space before the parenthesis and space before the angle bracket)
          • l.69, 72, 75: nit: spacing
          • l.72, 75, 87: nit: have one empty line between methods

          (JobHistoryFileReplayMapperV1.java)

          • l.41: wildcard imports are discouraged
          • l.70: nit: spacing
          • l.90: this comment does not apply to v.1 (this is specific to v.2)
          • l.102: nit: need a space at the end of "... for job"

          (SimpleEntityWriterV1.java)

          • l.61: alphaNums can be private; also recommend upper-case name (ALPHA_NUMS)
          • l.72-74: none of these variables may be used in v.1 (although they are needed in v.2); could you please check with your original patch and see if these are needed? if not, please remove them
          Show
          sjlee0 Sangjin Lee added a comment - It looks pretty good! These are mostly minor comments. Hopefully it won't be too hard to address them. Also, do take a look at the whitespace warning to clear up the jenkins. (JobHistoryFileReplayHelper.java) l.37: the class doesn't need to be public l.46: JOB_ID_PARSER can be private l.48: FileType can be private l.53: nit: spacing (no space before the parenthesis and space before the angle bracket) l.69, 72, 75: nit: spacing l.72, 75, 87: nit: have one empty line between methods (JobHistoryFileReplayMapperV1.java) l.41: wildcard imports are discouraged l.70: nit: spacing l.90: this comment does not apply to v.1 (this is specific to v.2) l.102: nit: need a space at the end of "... for job" (SimpleEntityWriterV1.java) l.61: alphaNums can be private; also recommend upper-case name (ALPHA_NUMS) l.72-74: none of these variables may be used in v.1 (although they are needed in v.2); could you please check with your original patch and see if these are needed? if not, please remove them
          Hide
          sjlee0 Sangjin Lee added a comment -

          Also, please test all the modes (with a pseudo-distributed cluster) to make sure they work as expected.

          Show
          sjlee0 Sangjin Lee added a comment - Also, please test all the modes (with a pseudo-distributed cluster) to make sure they work as expected.
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee Thanks a lot for quick review! I have updated my patch according to your suggestions. I have also test different modes in my single node cluster. I also test the scenario where conf or jobhistory file missing. As for the whitespace issues, they are introduced by existing code in MapredTestDriver, I am not sure if I should introduce more change in that file. But I provide two versions of patches, one without fix for existing white space issue in MapredTestDriver and one does include the fix. Thanks!

          Show
          lichangleo Chang Li added a comment - Sangjin Lee Thanks a lot for quick review! I have updated my patch according to your suggestions. I have also test different modes in my single node cluster. I also test the scenario where conf or jobhistory file missing. As for the whitespace issues, they are introduced by existing code in MapredTestDriver, I am not sure if I should introduce more change in that file. But I provide two versions of patches, one without fix for existing white space issue in MapredTestDriver and one does include the fix. Thanks!
          Hide
          sjlee0 Sangjin Lee added a comment -

          Thanks Chang Li!

          If I have to nitpick just one last thing, JOB_ID_PARSER should be static (in addition to private), and so should ALPHA_NUMS.

          With those changes, I'd be +1. However, since I'm not a committer, my vote is not binding. You'd need to get a (review and) blessing from a Hadoop committer. Hopefully this set of reviews would be sufficient for a committer's +1. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - Thanks Chang Li ! If I have to nitpick just one last thing, JOB_ID_PARSER should be static ( in addition to private), and so should ALPHA_NUMS. With those changes, I'd be +1. However, since I'm not a committer, my vote is not binding. You'd need to get a (review and) blessing from a Hadoop committer. Hopefully this set of reviews would be sufficient for a committer's +1. Thanks!
          Hide
          lichangleo Chang Li added a comment -

          Thanks so much for all the very helpful and patient reviews Sangjin Lee! I have done the static change in the latest patches. I also learned a lot from your work and reviews, thanks a lot !

          Show
          lichangleo Chang Li added a comment - Thanks so much for all the very helpful and patient reviews Sangjin Lee ! I have done the static change in the latest patches. I also learned a lot from your work and reviews, thanks a lot !
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 5m 28s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 33s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 13s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 mapreduce tests 106m 20s Tests passed in hadoop-mapreduce-client-jobclient.
              122m 53s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 7817674
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8129/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8129/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/8129/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 5m 28s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 33s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 13s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 mapreduce tests 106m 20s Tests passed in hadoop-mapreduce-client-jobclient.     122m 53s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 7817674 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/8129/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8129/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8129/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Jonathan Eagles could you please help review the latest patch? Thanks!

          Show
          lichangleo Chang Li added a comment - Jonathan Eagles could you please help review the latest patch? Thanks!
          Hide
          lichangleo Chang Li added a comment -

          Zhijie Shen, Junping Du could you please help review the latest patch? Thanks!

          Show
          lichangleo Chang Li added a comment - Zhijie Shen , Junping Du could you please help review the latest patch? Thanks!
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee, could you help commit this patch if your previous +1 still apply? Thanks!

          Show
          lichangleo Chang Li added a comment - Sangjin Lee , could you help commit this patch if your previous +1 still apply? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 21s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 9m 46s There were no new javac warning messages.
          -1 release audit 0m 19s The applied patch generated 1 release audit warnings.
          +1 checkstyle 0m 48s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 57s mvn install still works.
          +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse.
          +1 findbugs 0m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 99m 26s Tests failed in hadoop-mapreduce-client-jobclient.
              120m 17s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestReduceFetch
            hadoop.mapred.TestYARNRunner
            hadoop.mapred.TestClusterMRNotification
            hadoop.mapred.TestJavaSerialization
            hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 08f5de1
          Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9455/artifact/patchprocess/patchReleaseAuditProblems.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9455/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9455/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9455/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 9m 46s There were no new javac warning messages. -1 release audit 0m 19s The applied patch generated 1 release audit warnings. +1 checkstyle 0m 48s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 57s mvn install still works. +1 eclipse:eclipse 0m 38s The patch built with eclipse:eclipse. +1 findbugs 0m 56s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 99m 26s Tests failed in hadoop-mapreduce-client-jobclient.     120m 17s   Reason Tests Failed unit tests hadoop.mapred.TestReduceFetch   hadoop.mapred.TestYARNRunner   hadoop.mapred.TestClusterMRNotification   hadoop.mapred.TestJavaSerialization   hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 08f5de1 Release Audit https://builds.apache.org/job/PreCommit-YARN-Build/9455/artifact/patchprocess/patchReleaseAuditProblems.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9455/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9455/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9455/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          I just rekicked the jenkins run to see if the failures are transient.

          Show
          sjlee0 Sangjin Lee added a comment - I just rekicked the jenkins run to see if the failures are transient.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 23s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 8m 16s There were no new javac warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 36s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 34s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 0m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 100m 18s Tests failed in hadoop-mapreduce-client-jobclient.
              118m 55s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 7a98d94
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9458/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9458/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9458/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 23s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 8m 16s There were no new javac warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 36s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 34s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 50s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 100m 18s Tests failed in hadoop-mapreduce-client-jobclient.     118m 55s   Reason Tests Failed unit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12736208/YARN-2556.14.whitespacefix.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 7a98d94 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9458/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9458/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9458/console This message was automatically generated.
          Hide
          lichangleo Chang Li added a comment -

          Sangjin Lee, the failed unit test is related to MAPREDUCE-6508

          Show
          lichangleo Chang Li added a comment - Sangjin Lee , the failed unit test is related to MAPREDUCE-6508
          Hide
          sjlee0 Sangjin Lee added a comment -

          The latest patch is almost ready. I only have one small comment.

          (MapredTestDriver.java)

          • l.95: typo: "A job that launch ..." -> "A job that launches ..."

          If you could also just double check all modes work fine against the latest trunk, that would be great. Thanks!

          Show
          sjlee0 Sangjin Lee added a comment - The latest patch is almost ready. I only have one small comment. (MapredTestDriver.java) l.95: typo: "A job that launch ..." -> "A job that launches ..." If you could also just double check all modes work fine against the latest trunk, that would be great. Thanks!
          Hide
          lichangleo Chang Li added a comment -

          Thanks Sangjin Lee for more review! I have fixed the typo in .15 patch. Moreover, I have test the two mapper types(simple entity write mapper and jobhistory files replay mapper) on my single node cluster. Those tests all went well.

          Show
          lichangleo Chang Li added a comment - Thanks Sangjin Lee for more review! I have fixed the typo in .15 patch. Moreover, I have test the two mapper types(simple entity write mapper and jobhistory files replay mapper) on my single node cluster. Those tests all went well.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 35s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 7 new or modified test files.
          +1 javac 7m 53s There were no new javac warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 33s There were no new checkstyle issues.
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 31s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 98m 7s Tests failed in hadoop-mapreduce-client-jobclient.
              116m 28s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12767126/YARN-2556.15.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 79b8d60
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9466/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9466/testReport/
          Java 1.7.0_55
          uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/9466/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 35s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 7 new or modified test files. +1 javac 7m 53s There were no new javac warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 33s There were no new checkstyle issues. +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 31s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 98m 7s Tests failed in hadoop-mapreduce-client-jobclient.     116m 28s   Reason Tests Failed unit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12767126/YARN-2556.15.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 79b8d60 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-YARN-Build/9466/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/9466/testReport/ Java 1.7.0_55 uname Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/9466/console This message was automatically generated.
          Hide
          sjlee0 Sangjin Lee added a comment -

          The latest patch (v.15) looks good to me. I'll commit it shortly.

          Show
          sjlee0 Sangjin Lee added a comment - The latest patch (v.15) looks good to me. I'll commit it shortly.
          Hide
          sjlee0 Sangjin Lee added a comment -

          Committed it to trunk and branch-2 respectively. Thanks Chang Li for your valuable contribution and patience!

          Show
          sjlee0 Sangjin Lee added a comment - Committed it to trunk and branch-2 respectively. Thanks Chang Li for your valuable contribution and patience!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #8654 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8654/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #8654 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8654/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #560 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/560/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #560 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/560/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/545/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #545 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/545/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2445 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2445/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2445 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2445/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #1281 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1281/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #1281 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1281/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2494 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2494/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2494 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2494/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/508/)
          YARN-2556. Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #508 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/508/ ) YARN-2556 . Tool to measure the performance of the timeline server (Chang (sjlee: rev 58590fef49bf45fc97c81277560e08da6b753f95) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayHelper.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineEntityConverterV1.java hadoop-yarn-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/SimpleEntityWriterV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileReplayMapperV1.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/JobHistoryFileParser.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TimelineServicePerformance.java
          Hide
          Naganarasimha Naganarasimha G R added a comment -

          Hi Sangjin Lee & Chang Li
          In Some Forum User Query came across an issue where each Insert was taking more time when the existing Level DB data size was more(about 3GB disk size), continuously at the rate of 10 inserts per sec for span of 15 mins . During each Insertion, query is happening to pick the Date for identifying CreationTime which i presume to be the reason for ATS inserts being slow. May be we can optimize the test to have some initial Level DB Data and then test the performance. This would also be useful to evaluate the performance of ATSV1.5. Thoughts ?

          Show
          Naganarasimha Naganarasimha G R added a comment - Hi Sangjin Lee & Chang Li In Some Forum User Query came across an issue where each Insert was taking more time when the existing Level DB data size was more(about 3GB disk size), continuously at the rate of 10 inserts per sec for span of 15 mins . During each Insertion, query is happening to pick the Date for identifying CreationTime which i presume to be the reason for ATS inserts being slow. May be we can optimize the test to have some initial Level DB Data and then test the performance. This would also be useful to evaluate the performance of ATSV1.5. Thoughts ?
          Hide
          xgong Xuan Gong added a comment - - edited

          Thanks for the work.
          Sangjin Lee and Chang Li Could you give us some instruction on how to run this performance tool ? Maybe add a document on related ats docs and at least give us an example command to run this tool.

          Show
          xgong Xuan Gong added a comment - - edited Thanks for the work. Sangjin Lee and Chang Li Could you give us some instruction on how to run this performance tool ? Maybe add a document on related ats docs and at least give us an example command to run this tool.
          Hide
          lichangleo Chang Li added a comment -

          Hi Xuan Gong, here is the usage print out by the tool

           
          Usage: [-m <maps>] number of mappers (default: 1)
               [-v] timeline service version
               [-mtype <mapper type in integer>]
                    1. simple entity write mapper
                    2. jobhistory files replay mapper
               [-s <(KBs)test>] number of KB per put (mtype=1, default: 1 KB)
               [-t] package sending iterations per mapper (mtype=1, default: 100)
               [-d <path>] root path of job history files (mtype=2)
               [-r <replay mode>] (mtype=2)
                    1. write all entities for a job in one put (default)
                    2. write one entity at a time

          there are two different modes to test, one is simple entity writer, where each mapper create your specified size of entities and put them to timeline server. The other mode of test is by replaying jobhistory files, which offer a more realistic test. In the case of jobhistory file replay test, you put testing jobhistory files(both the job history file and job conf file) under a directory, and then you specify the testing dir by -d option. You specify the test mode by -mtype option.
          Right now the usage won't get printed out if you pass no options, but only print out when you pass the wrong options. When you give no parameters, the test run with simple entity write mode and default setting. So maybe we want to print out this usage if we don't pass any parameter?

          Show
          lichangleo Chang Li added a comment - Hi Xuan Gong , here is the usage print out by the tool Usage: [-m <maps>] number of mappers ( default : 1) [-v] timeline service version [-mtype <mapper type in integer>] 1. simple entity write mapper 2. jobhistory files replay mapper [-s <(KBs)test>] number of KB per put (mtype=1, default : 1 KB) [-t] package sending iterations per mapper (mtype=1, default : 100) [-d <path>] root path of job history files (mtype=2) [-r <replay mode>] (mtype=2) 1. write all entities for a job in one put ( default ) 2. write one entity at a time there are two different modes to test, one is simple entity writer, where each mapper create your specified size of entities and put them to timeline server. The other mode of test is by replaying jobhistory files, which offer a more realistic test. In the case of jobhistory file replay test, you put testing jobhistory files(both the job history file and job conf file) under a directory, and then you specify the testing dir by -d option. You specify the test mode by -mtype option. Right now the usage won't get printed out if you pass no options, but only print out when you pass the wrong options. When you give no parameters, the test run with simple entity write mode and default setting. So maybe we want to print out this usage if we don't pass any parameter?
          Hide
          lichangleo Chang Li added a comment -

          Thanks Naganarasimha G R for suggesting optimization! +1 on the idea of creating some initial leveldb data before test the performance. Create YARN-4339 to work on this idea.

          Show
          lichangleo Chang Li added a comment - Thanks Naganarasimha G R for suggesting optimization! +1 on the idea of creating some initial leveldb data before test the performance. Create YARN-4339 to work on this idea.
          Hide
          lichangleo Chang Li added a comment -

          create YARN-4341 to track work of add doc about timeline performance tool usage

          Show
          lichangleo Chang Li added a comment - create YARN-4341 to track work of add doc about timeline performance tool usage
          Hide
          sjlee0 Sangjin Lee added a comment -

          +1 with the proposal to add documentation. The command line help is useful, but it would be good to have a little more detail in the documentation.

          Show
          sjlee0 Sangjin Lee added a comment - +1 with the proposal to add documentation. The command line help is useful, but it would be good to have a little more detail in the documentation.

            People

            • Assignee:
              lichangleo Chang Li
              Reporter:
              jeagles Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development