Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6622

Add capability to set JHS job cache to a task-based limit

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property:
      1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB.
      2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.
      Show
      Two recommendations for the mapreduce.jobhistory.loadedtasks.cache.size property: 1) For every 100k of cache size, set the heap size of the Job History Server to 1.2GB. For example, mapreduce.jobhistory.loadedtasks.cache.size=500000, heap size=6GB. 2) Make sure that the cache size is larger than the number of tasks required for the largest job run on the cluster. It might be a good idea to set the value slightly higher (say, 20%) in order to allow for job size growth.

      Description

      When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs can be of varying size. This is generally not a problem when the jobs sizes are uniform or small, but when the job sizes can be very large (say greater than 250k tasks), then the JHS heap size can grow tremendously.

      In cases, where multiple jobs are very large, then the JHS can lock up and spend all its time in GC. However, since the cache is holding on to all the jobs, not much heap space can be freed up.

      By setting a property that sets a cap on the number of tasks allowed in the cache and since the total number of tasks loaded is directly proportional to the amount of heap used, this should help prevent the JHS from locking up.

      1. MAPREDUCE-6622.001.patch
        14 kB
        Ray Chiang
      2. MAPREDUCE-6622.002.patch
        15 kB
        Ray Chiang
      3. MAPREDUCE-6622.003.patch
        15 kB
        Ray Chiang
      4. MAPREDUCE-6622.004.patch
        16 kB
        Ray Chiang
      5. MAPREDUCE-6622.005.patch
        17 kB
        Ray Chiang
      6. MAPREDUCE-6622.006.patch
        17 kB
        Ray Chiang
      7. MAPREDUCE-6622.007.patch
        17 kB
        Ray Chiang
      8. MAPREDUCE-6622.008.patch
        17 kB
        Ray Chiang
      9. MAPREDUCE-6622.009.patch
        20 kB
        Ray Chiang
      10. MAPREDUCE-6622.010.patch
        19 kB
        Ray Chiang
      11. MAPREDUCE-6622.011.patch
        20 kB
        Ray Chiang
      12. MAPREDUCE-6622.012.patch
        23 kB
        Ray Chiang
      13. MAPREDUCE-6622.013.patch
        23 kB
        Ray Chiang
      14. MAPREDUCE-6622.014.patch
        23 kB
        Robert Kanter

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 45s Maven dependency ordering for branch
        +1 mvninstall 6m 50s trunk passed
        +1 compile 1m 27s trunk passed with JDK v1.8.0_66
        +1 compile 1m 44s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 19s trunk passed
        +1 mvneclipse 0m 36s trunk passed
        +1 findbugs 2m 38s trunk passed
        +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 6s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 23s Maven dependency ordering for patch
        +1 mvninstall 1m 3s the patch passed
        +1 compile 1m 35s the patch passed with JDK v1.8.0_66
        +1 javac 1m 35s the patch passed
        +1 compile 1m 43s the patch passed with JDK v1.7.0_91
        +1 javac 1m 43s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 3m 12s the patch passed
        +1 javadoc 0m 47s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 1s the patch passed with JDK v1.7.0_91
        +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 38s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66.
        -1 unit 49m 6s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_66.
        +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 54s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91.
        -1 unit 49m 14s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_91.
        -1 asflicense 0m 20s Patch generated 14 ASF License warnings.
        135m 50s



        Reason Tests
        JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.v2.hs.TestJobHistory
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryServer
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
        JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.mapreduce.v2.hs.TestJobHistory
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryServer
          org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785110/MAPREDUCE-6622.001.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux fb87bec00d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / df99ea8
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_66.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_91.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_91.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 45s Maven dependency ordering for branch +1 mvninstall 6m 50s trunk passed +1 compile 1m 27s trunk passed with JDK v1.8.0_66 +1 compile 1m 44s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 36s trunk passed +1 findbugs 2m 38s trunk passed +1 javadoc 1m 3s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 6s trunk passed with JDK v1.7.0_91 0 mvndep 0m 23s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 1m 35s the patch passed with JDK v1.8.0_66 +1 javac 1m 35s the patch passed +1 compile 1m 43s the patch passed with JDK v1.7.0_91 +1 javac 1m 43s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 12s the patch passed +1 javadoc 0m 47s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 1s the patch passed with JDK v1.7.0_91 +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 0m 38s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66. -1 unit 49m 6s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_66. +1 unit 2m 30s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 0m 54s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91. -1 unit 49m 14s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 20s Patch generated 14 ASF License warnings. 135m 50s Reason Tests JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapreduce.v2.hs.TestJobHistory   org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryServer   org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents JDK v1.7.0_91 Timed out junit tests org.apache.hadoop.mapreduce.v2.hs.TestJobHistory   org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryServer   org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785110/MAPREDUCE-6622.001.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux fb87bec00d55 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / df99ea8 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6297/console This message was automatically generated.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the patch, Ray!

        The documentation for the existing property for limiting based on job count needs to be updated to mention it is ignored if the new property is set.

        Seems like it would be straightforward to pull out old jobs until we've freed up enough tasks to pay for the job trying to be added to the cache. Now we have a background thread with a new property to configure how often it cleans – does the cache blow way out of proportion if we don't clean up fast enough (e.g.: history server gets programmatically hammered for many jobs)? Adding yet another guava dependency isn't appealing unless really necessary. If we are sticking with the guava cache, is it essential to call cleanUp in a background thread or won't this cleanup automatically happen as new jobs are loaded into the cache?

        Show
        jlowe Jason Lowe added a comment - Thanks for the patch, Ray! The documentation for the existing property for limiting based on job count needs to be updated to mention it is ignored if the new property is set. Seems like it would be straightforward to pull out old jobs until we've freed up enough tasks to pay for the job trying to be added to the cache. Now we have a background thread with a new property to configure how often it cleans – does the cache blow way out of proportion if we don't clean up fast enough (e.g.: history server gets programmatically hammered for many jobs)? Adding yet another guava dependency isn't appealing unless really necessary. If we are sticking with the guava cache, is it essential to call cleanUp in a background thread or won't this cleanup automatically happen as new jobs are loaded into the cache?
        Hide
        rchiang Ray Chiang added a comment -

        The documentation for the existing property for limiting based on job count needs to be updated to mention it is ignored if the new property is set.

        Will do.

        Seems like it would be straightforward to pull out old jobs until we've freed up enough tasks to pay for the job trying to be added to the cache. Now we have a background thread with a new property to configure how often it cleans – does the cache blow way out of proportion if we don't clean up fast enough (e.g.: history server gets programmatically hammered for many jobs)? Adding yet another guava dependency isn't appealing unless really necessary. If we are sticking with the guava cache, is it essential to call cleanUp in a background thread or won't this cleanup automatically happen as new jobs are loaded into the cache?

        I ended up having to call cleanUp() in order to get the unit tests to pass, but those admittedly run in a very short amount of time. There's definitely some lack of determinism in that size() returns an approximate size (according to the documentation). I'd say my biggest concern is that GC without any explicit cache churn (i.e. users clicking on job links) won't force the cache to explicitly cleanup for when you have several really large jobs, but the background thread would.

        I could change the cache setting to mean:

        • -1 Never call cleanUp() explicitly
        • 0 Always call cleanUp() explicitly after each write
        • >0 Run in cleanUp() in a periodic thread

        And I didn't like the move to Guava either, but on the bright side, it looks like Java 8 ConcurrentHashMap+lambdas can mimic a basic cache. Some of the less sophisticated usages can move to that when JDK8 becomes the baseline.

        Show
        rchiang Ray Chiang added a comment - The documentation for the existing property for limiting based on job count needs to be updated to mention it is ignored if the new property is set. Will do. Seems like it would be straightforward to pull out old jobs until we've freed up enough tasks to pay for the job trying to be added to the cache. Now we have a background thread with a new property to configure how often it cleans – does the cache blow way out of proportion if we don't clean up fast enough (e.g.: history server gets programmatically hammered for many jobs)? Adding yet another guava dependency isn't appealing unless really necessary. If we are sticking with the guava cache, is it essential to call cleanUp in a background thread or won't this cleanup automatically happen as new jobs are loaded into the cache? I ended up having to call cleanUp() in order to get the unit tests to pass, but those admittedly run in a very short amount of time. There's definitely some lack of determinism in that size() returns an approximate size (according to the documentation). I'd say my biggest concern is that GC without any explicit cache churn (i.e. users clicking on job links) won't force the cache to explicitly cleanup for when you have several really large jobs, but the background thread would. I could change the cache setting to mean: -1 Never call cleanUp() explicitly 0 Always call cleanUp() explicitly after each write >0 Run in cleanUp() in a periodic thread And I didn't like the move to Guava either, but on the bright side, it looks like Java 8 ConcurrentHashMap+lambdas can mimic a basic cache. Some of the less sophisticated usages can move to that when JDK8 becomes the baseline.
        Hide
        jlowe Jason Lowe added a comment -

        My main point was that I don't think it would be hard to avoid both the undesirable extra Guava dependency and not-quite-deterministic behavior of the cache. Seems like it would be pretty straightforward to maintain the cache ourselves with the LinkedHashMap to help track what hasn't been used recently. Did you look into that approach and abandon it?

        Show
        jlowe Jason Lowe added a comment - My main point was that I don't think it would be hard to avoid both the undesirable extra Guava dependency and not-quite-deterministic behavior of the cache. Seems like it would be pretty straightforward to maintain the cache ourselves with the LinkedHashMap to help track what hasn't been used recently. Did you look into that approach and abandon it?
        Hide
        rkanter Robert Kanter added a comment -

        If we add the -1, 0, and >0 for the cleanUp(), make sure to explain the implications of that in the description.

        A few other small things:

        1. We should change if (loadedTasksCacheSize==-1) { to if (loadedTasksCacheSize<=-1) {. Otherwise the user will get an IllegalArgumentException when it tries to do the CacheBuilder stuff. This way, it will revert to the old behavior, which is nicer.
        2. Can we call lruJobTracker something else? JobTracker is used enough in Hadoop 1
        Show
        rkanter Robert Kanter added a comment - If we add the -1, 0, and >0 for the cleanUp(), make sure to explain the implications of that in the description. A few other small things: We should change if (loadedTasksCacheSize==-1) { to if (loadedTasksCacheSize<=-1) { . Otherwise the user will get an IllegalArgumentException when it tries to do the CacheBuilder stuff. This way, it will revert to the old behavior, which is nicer. Can we call lruJobTracker something else? JobTracker is used enough in Hadoop 1
        Hide
        rchiang Ray Chiang added a comment -

        Oh, I see.

        I did try the simplest approach of overriding LinkedHashMap#removeEldestEntry(), but an alternate measure like Cache weight (or in this specific case Task Count) didn't update. The method seems to allow the CachedHistoryStorage member variables to be visible within the method, but not modifiable.

        I admittedly did not try something more sophisticated with LinkedHashMap before jumping to the Guava cache implementation.

        Show
        rchiang Ray Chiang added a comment - Oh, I see. I did try the simplest approach of overriding LinkedHashMap#removeEldestEntry(), but an alternate measure like Cache weight (or in this specific case Task Count) didn't update. The method seems to allow the CachedHistoryStorage member variables to be visible within the method, but not modifiable. I admittedly did not try something more sophisticated with LinkedHashMap before jumping to the Guava cache implementation.
        Hide
        rchiang Ray Chiang added a comment -

        We should change if (loadedTasksCacheSize==-1) { to if (loadedTasksCacheSize<=-1) {. Otherwise the user will get an IllegalArgumentException when it tries to do the CacheBuilder stuff. This way, it will revert to the old behavior, which is nicer.

        Will do.

        Can we call lruJobTracker something else? JobTracker is used enough in Hadoop 1

        Good point. I used a clearly different variable name while trying out different implementations. I could switch back to the old name of loadedJobCache.

        Show
        rchiang Ray Chiang added a comment - We should change if (loadedTasksCacheSize==-1) { to if (loadedTasksCacheSize<=-1) {. Otherwise the user will get an IllegalArgumentException when it tries to do the CacheBuilder stuff. This way, it will revert to the old behavior, which is nicer. Will do. Can we call lruJobTracker something else? JobTracker is used enough in Hadoop 1 Good point. I used a clearly different variable name while trying out different implementations. I could switch back to the old name of loadedJobCache.
        Hide
        kasha Karthik Kambatla added a comment -

        I have used the guava cache before (can't remember where), and have found it to be very useful. Except for the compatibility concerns in guava, I don't mind us using it on the server side at all. The client is a different story, but I don't think we need to worry about that here. Jason Lowe, Ray Chiang - did you guys have any other specific concerns with using guava that I am discounting?

        I ended up having to call cleanUp() in order to get the unit tests to pass, but those admittedly run in a very short amount of time.

        I don't see the need to do cleanups to ensure the unit tests pass.

        On how often to clean up, the cache considers eviction on every load. If this is going to be a cache with frequent loads, we don't need to have another thread doing the cleanup. Ray Chiang - in your test, did you consider continuously loading new jobs? If no new jobs are loaded for a while, the likelihood of the cache cleaning up is low. Even if it does, it will be on read. But, do we need to evict jobs if we are not loading any new ones?

        Show
        kasha Karthik Kambatla added a comment - I have used the guava cache before (can't remember where), and have found it to be very useful. Except for the compatibility concerns in guava, I don't mind us using it on the server side at all. The client is a different story, but I don't think we need to worry about that here. Jason Lowe , Ray Chiang - did you guys have any other specific concerns with using guava that I am discounting? I ended up having to call cleanUp() in order to get the unit tests to pass, but those admittedly run in a very short amount of time. I don't see the need to do cleanups to ensure the unit tests pass. On how often to clean up, the cache considers eviction on every load. If this is going to be a cache with frequent loads, we don't need to have another thread doing the cleanup. Ray Chiang - in your test, did you consider continuously loading new jobs? If no new jobs are loaded for a while, the likelihood of the cache cleaning up is low. Even if it does, it will be on read. But, do we need to evict jobs if we are not loading any new ones?
        Hide
        kasha Karthik Kambatla added a comment -

        If we do decide to continue with using the guava cache, I would like to review the patch before it gets committed.

        Show
        kasha Karthik Kambatla added a comment - If we do decide to continue with using the guava cache, I would like to review the patch before it gets committed.
        Hide
        rchiang Ray Chiang added a comment -
        • Update documentation in mapred-default.xml
        • Update behavior of cache sleep property
        • Fix cache variable name
        • Make value checking for loadedtasks property more robust
        Show
        rchiang Ray Chiang added a comment - Update documentation in mapred-default.xml Update behavior of cache sleep property Fix cache variable name Make value checking for loadedtasks property more robust
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 24s Maven dependency ordering for branch
        +1 mvninstall 6m 34s trunk passed
        +1 compile 1m 31s trunk passed with JDK v1.8.0_66
        +1 compile 1m 42s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 21s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 2m 34s trunk passed
        +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66
        +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91
        0 mvndep 0m 25s Maven dependency ordering for patch
        +1 mvninstall 1m 2s the patch passed
        +1 compile 1m 23s the patch passed with JDK v1.8.0_66
        +1 javac 1m 23s the patch passed
        +1 compile 1m 38s the patch passed with JDK v1.7.0_91
        +1 javac 1m 38s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 1m 13s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 4s the patch passed
        +1 javadoc 0m 47s the patch passed with JDK v1.8.0_66
        +1 javadoc 1m 0s the patch passed with JDK v1.7.0_91
        +1 unit 1m 48s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 0m 36s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66.
        +1 unit 5m 30s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 7s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91.
        +1 unit 5m 52s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_91.
        -1 asflicense 0m 18s Patch generated 14 ASF License warnings.
        47m 7s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785337/MAPREDUCE-6622.002.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 15264aa1c480 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 772ea7b
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 24s Maven dependency ordering for branch +1 mvninstall 6m 34s trunk passed +1 compile 1m 31s trunk passed with JDK v1.8.0_66 +1 compile 1m 42s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 21s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 34s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_91 0 mvndep 0m 25s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 1m 23s the patch passed with JDK v1.8.0_66 +1 javac 1m 23s the patch passed +1 compile 1m 38s the patch passed with JDK v1.7.0_91 +1 javac 1m 38s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 4s the patch passed +1 javadoc 0m 47s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 0s the patch passed with JDK v1.7.0_91 +1 unit 1m 48s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 0m 36s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_66. +1 unit 5m 30s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_66. +1 unit 2m 7s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_91. +1 unit 5m 52s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 18s Patch generated 14 ASF License warnings. 47m 7s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12785337/MAPREDUCE-6622.002.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 15264aa1c480 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 772ea7b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6301/console This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: ASF license warning

        Same set of files as earlier. None of the files come from the patch.

        Show
        rchiang Ray Chiang added a comment - RE: ASF license warning Same set of files as earlier. None of the files come from the patch.
        Hide
        rchiang Ray Chiang added a comment -

        While I do have some doubts about general Guava library compatibility, I don't have a lot with their Cache library. While I was trying to figure out how to implement this, I wrote up some separate code using the latest (v19) and the same API calls worked fine in the upstream tree (v11).

        Show
        rchiang Ray Chiang added a comment - While I do have some doubts about general Guava library compatibility, I don't have a lot with their Cache library. While I was trying to figure out how to implement this, I wrote up some separate code using the latest (v19) and the same API calls worked fine in the upstream tree (v11).
        Hide
        kasha Karthik Kambatla added a comment -

        Sorry for the delay in getting to this. Took a quick look at the patch, some high-level comments:

        1. Since this is the JobHistoryServer and no client code runs in it, I am comfortable with us using guava cache. I am aware of Ray's attempts at doing the same using the previous LinkedHashMap and it was fairly involved. I ll let him comment on the details here.
        2. Since the cache automatically evicts jobs when it loads a new job, I don't see the need for the cleanup thread. Having an additional cleanup thread kind of goes against the reason we are using guava cache in the first place, simplicity. We could choose to use time-based eviction if we want, there is also a "ticker" we could use to test it.
        3. Is okay to not honor loadedjobs.cache.size when loadedtasks.cache.size is specified? Jason Lowe - do you remember if the number of jobs was being used as a proxy for the memory usage?
        4. What happens when a job with tasks more than the allowed value needs to be loaded? Can we add a test to verify that case?
        Show
        kasha Karthik Kambatla added a comment - Sorry for the delay in getting to this. Took a quick look at the patch, some high-level comments: Since this is the JobHistoryServer and no client code runs in it, I am comfortable with us using guava cache. I am aware of Ray's attempts at doing the same using the previous LinkedHashMap and it was fairly involved. I ll let him comment on the details here. Since the cache automatically evicts jobs when it loads a new job, I don't see the need for the cleanup thread. Having an additional cleanup thread kind of goes against the reason we are using guava cache in the first place, simplicity. We could choose to use time-based eviction if we want, there is also a "ticker" we could use to test it. Is okay to not honor loadedjobs.cache.size when loadedtasks.cache.size is specified? Jason Lowe - do you remember if the number of jobs was being used as a proxy for the memory usage? What happens when a job with tasks more than the allowed value needs to be loaded? Can we add a test to verify that case?
        Hide
        rchiang Ray Chiang added a comment -

        For LinkedHashMap implementations, I have tried the following:

        A. I first did an attempt at updating LinkedHashMap by just modifying removeEldestEntry(). That didn't work.

        B. I just did a second attempt this morning at extending LinkedHashMap and adding the ability to track the total task count. It tracks the task count correctly, but removeEldestEntry() only gets triggered once per put. I don't see a good way to improve on that.

        Show
        rchiang Ray Chiang added a comment - For LinkedHashMap implementations, I have tried the following: A. I first did an attempt at updating LinkedHashMap by just modifying removeEldestEntry(). That didn't work. B. I just did a second attempt this morning at extending LinkedHashMap and adding the ability to track the total task count. It tracks the task count correctly, but removeEldestEntry() only gets triggered once per put. I don't see a good way to improve on that.
        Hide
        rchiang Ray Chiang added a comment -
        • Remove references to using Cache#cleanUp()
        • Add new unit test for adding a job that is larger than the set task limit
        Show
        rchiang Ray Chiang added a comment - Remove references to using Cache#cleanUp() Add new unit test for adding a job that is larger than the set task limit
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 7m 5s trunk passed
        +1 compile 1m 33s trunk passed with JDK v1.8.0_72
        +1 compile 1m 44s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 2m 26s trunk passed
        +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 5s the patch passed
        +1 compile 1m 36s the patch passed with JDK v1.8.0_72
        +1 javac 1m 36s the patch passed
        +1 compile 1m 46s the patch passed with JDK v1.7.0_95
        +1 javac 1m 46s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 1m 16s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 3m 10s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95
        +1 unit 1m 50s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 37s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        +1 unit 5m 28s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72.
        +1 unit 2m 10s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        +1 unit 5m 54s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 21s Patch generated 14 ASF License warnings.
        48m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789043/MAPREDUCE-6622.003.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux df4befa3b9ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3fab885
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 5s trunk passed +1 compile 1m 33s trunk passed with JDK v1.8.0_72 +1 compile 1m 44s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 26s trunk passed +1 javadoc 0m 58s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 7s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 1m 36s the patch passed with JDK v1.8.0_72 +1 javac 1m 36s the patch passed +1 compile 1m 46s the patch passed with JDK v1.7.0_95 +1 javac 1m 46s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 10s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95 +1 unit 1m 50s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. +1 unit 0m 37s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. +1 unit 5m 28s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72. +1 unit 2m 10s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. +1 unit 5m 54s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 21s Patch generated 14 ASF License warnings. 48m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789043/MAPREDUCE-6622.003.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux df4befa3b9ae 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3fab885 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6338/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: ASF license warnings

        ASF license issues not related to patch.

        Show
        rchiang Ray Chiang added a comment - RE: ASF license warnings ASF license issues not related to patch.
        Hide
        rkanter Robert Kanter added a comment -

        Looks good overall, some minor things:

        • The release note recommendations for mapreduce.jobhistory.loadedtasks.cache.size are good, but I think we should also put them into mapred-default.xml.
        • Undo the blah.blah.* imports
        • Add a comment explaining why you set the concurrency levels to 1. I can easily see someone coming along later and saying, "hey, more concurrency sounds good" without realizing the funny side effects that you showed me offline
        • Add a comment explaining why you do the -1 in Math.min(loadedTasksCacheSize-1, value.getTotalMaps() + value.getTotalReduces()); for the weights
        • Use conf.setInt(..., #) instead of conf.set(..., "#") in the tests.
        Show
        rkanter Robert Kanter added a comment - Looks good overall, some minor things: The release note recommendations for mapreduce.jobhistory.loadedtasks.cache.size are good, but I think we should also put them into mapred-default.xml. Undo the blah.blah.* imports Add a comment explaining why you set the concurrency levels to 1. I can easily see someone coming along later and saying, "hey, more concurrency sounds good" without realizing the funny side effects that you showed me offline Add a comment explaining why you do the -1 in Math.min(loadedTasksCacheSize-1, value.getTotalMaps() + value.getTotalReduces()); for the weights Use conf.setInt(..., #) instead of conf.set(..., "#") in the tests.
        Hide
        rchiang Ray Chiang added a comment -

        Update patch based on Robert's feedback

        • Copy Release Notes into mapred-default.xml comments
        • Remove "*" imports
        • Add comment about Guava Cache concurrency level setting
        • Remove -1 from cache size. Leftover legacy from earlier testing.
        • Change configuration calls to setInt()
        Show
        rchiang Ray Chiang added a comment - Update patch based on Robert's feedback Copy Release Notes into mapred-default.xml comments Remove "*" imports Add comment about Guava Cache concurrency level setting Remove -1 from cache size. Leftover legacy from earlier testing. Change configuration calls to setInt()
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 7m 47s trunk passed
        +1 compile 2m 16s trunk passed with JDK v1.8.0_72
        +1 compile 2m 1s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 25s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 2m 32s trunk passed
        +1 javadoc 1m 15s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 10s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 11s the patch passed
        +1 compile 2m 13s the patch passed with JDK v1.8.0_72
        +1 javac 2m 13s the patch passed
        +1 compile 2m 0s the patch passed with JDK v1.7.0_95
        +1 javac 2m 0s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 1m 19s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 16s the patch passed
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 7s the patch passed with JDK v1.7.0_95
        +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 52s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        +1 unit 6m 29s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72.
        +1 unit 2m 24s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 51s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        +1 unit 6m 13s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 21s Patch generated 14 ASF License warnings.
        54m 18s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789079/MAPREDUCE-6622.004.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 576b1d5b719f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 66289a3
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/artifact/patchprocess/whitespace-eol.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 7m 47s trunk passed +1 compile 2m 16s trunk passed with JDK v1.8.0_72 +1 compile 2m 1s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 25s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 32s trunk passed +1 javadoc 1m 15s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 10s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 2m 13s the patch passed with JDK v1.8.0_72 +1 javac 2m 13s the patch passed +1 compile 2m 0s the patch passed with JDK v1.7.0_95 +1 javac 2m 0s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 34s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 7s the patch passed with JDK v1.7.0_95 +1 unit 2m 26s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. +1 unit 0m 52s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. +1 unit 6m 29s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72. +1 unit 2m 24s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 51s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. +1 unit 6m 13s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 21s Patch generated 14 ASF License warnings. 54m 18s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789079/MAPREDUCE-6622.004.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 576b1d5b719f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66289a3 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6341/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        LGTM +1

        Karthik Kambatla, Jason Lowe, any other comments?

        Show
        rkanter Robert Kanter added a comment - LGTM +1 Karthik Kambatla , Jason Lowe , any other comments?
        Hide
        kasha Karthik Kambatla added a comment -

        Comments on the latest patch:

        1. Unused variable cleanupThread left over from previous iterations.
        2. The description for loadedjobs.cache.size config says it is ignored if loadedtasks.cache.size is set. A default value of -1 actually means it is set. I like the previous approach of not setting it at all.
        3. Also, when reading the value for loadedtasks.cache.size, don't we want to enforce a minimum value for it? May be, 1?
        4. I am not sure I understand how entries are loaded into the cache. The CacheBuilder calls getFullJob, but the latter checks if the entry is present in the cache? If I understand it right, the public getFullJob() should just call loadedJobCache.get(jobID). The CacheLoader should call a private method loadJob(jobID). The current contents of getFullJob - get fileInfo etc. - should move to loadJob.
        Show
        kasha Karthik Kambatla added a comment - Comments on the latest patch: Unused variable cleanupThread left over from previous iterations. The description for loadedjobs.cache.size config says it is ignored if loadedtasks.cache.size is set. A default value of -1 actually means it is set. I like the previous approach of not setting it at all. Also, when reading the value for loadedtasks.cache.size, don't we want to enforce a minimum value for it? May be, 1? I am not sure I understand how entries are loaded into the cache. The CacheBuilder calls getFullJob, but the latter checks if the entry is present in the cache? If I understand it right, the public getFullJob() should just call loadedJobCache.get(jobID) . The CacheLoader should call a private method loadJob(jobID) . The current contents of getFullJob - get fileInfo etc. - should move to loadJob.
        Hide
        jlowe Jason Lowe added a comment -

        Sorry for the delay in responding, as I was out on vacation.

        Jason Lowe, Ray Chiang - did you guys have any other specific concerns with using guava that I am discounting?

        My main concern was that we were adding a separate thread and extra overhead for something that didn't seem worth the trouble. Now that the cache code has been significantly cleaned up, I'm more comfortable with going the guava route.

        Jason Lowe - do you remember if the number of jobs was being used as a proxy for the memory usage?

        Yes, it was essentially a proxy for capping memory usage. It's also a performance tunable, since caching more jobs can lead to improved performance depending upon client access patterns. However I think it is OK to override the old value with the new value as the behavior is clearly documented and we're not providing any default value that would override the user's old settings implicitly when they upgrade. Users would have to go out of their way to set this value and in doing so should encounter the documentation explaining the semantics when they set it.

        Show
        jlowe Jason Lowe added a comment - Sorry for the delay in responding, as I was out on vacation. Jason Lowe, Ray Chiang - did you guys have any other specific concerns with using guava that I am discounting? My main concern was that we were adding a separate thread and extra overhead for something that didn't seem worth the trouble. Now that the cache code has been significantly cleaned up, I'm more comfortable with going the guava route. Jason Lowe - do you remember if the number of jobs was being used as a proxy for the memory usage? Yes, it was essentially a proxy for capping memory usage. It's also a performance tunable, since caching more jobs can lead to improved performance depending upon client access patterns. However I think it is OK to override the old value with the new value as the behavior is clearly documented and we're not providing any default value that would override the user's old settings implicitly when they upgrade. Users would have to go out of their way to set this value and in doing so should encounter the documentation explaining the semantics when they set it.
        Hide
        rchiang Ray Chiang added a comment -
        • Remove unused member variable cleanUpThread
        • Change mapreduce.jobhistory.loadedtasks.cache.size default value in mapred-default.xml to empty
        • Add sanity checking for mapreduce.jobhistory.loadedtasks.cache.size property and only use the new cache if sanity checking passes
        Show
        rchiang Ray Chiang added a comment - Remove unused member variable cleanUpThread Change mapreduce.jobhistory.loadedtasks.cache.size default value in mapred-default.xml to empty Add sanity checking for mapreduce.jobhistory.loadedtasks.cache.size property and only use the new cache if sanity checking passes
        Hide
        rchiang Ray Chiang added a comment -

        In order to minimize impact, I didn't really do any real rewrite of the internals of CachedHistoryStorage. I just swapped in the cache access.

        Here's my understanding of getFullJob():

        • It's using the HistoryFileManager "fileInfo" as a proxy to determine whether the job exists.
        • If the job exists, it then checks the cache via loadedJobCache#getIfPresent()
        • If the job isn't in the cache, it loads the file into memory
        • If job history file has been deleted, getFullJob returns null
        Show
        rchiang Ray Chiang added a comment - In order to minimize impact, I didn't really do any real rewrite of the internals of CachedHistoryStorage. I just swapped in the cache access. Here's my understanding of getFullJob(): It's using the HistoryFileManager "fileInfo" as a proxy to determine whether the job exists. If the job exists, it then checks the cache via loadedJobCache#getIfPresent() If the job isn't in the cache, it loads the file into memory If job history file has been deleted, getFullJob returns null
        Hide
        rchiang Ray Chiang added a comment -
        • Fix documentation error in property.
        Show
        rchiang Ray Chiang added a comment - Fix documentation error in property.
        Hide
        rchiang Ray Chiang added a comment -
        • Further refactoring based on Karthik's feedback
        Show
        rchiang Ray Chiang added a comment - Further refactoring based on Karthik's feedback
        Hide
        rchiang Ray Chiang added a comment -
        • Keep forgetting to fix whitespace
        Show
        rchiang Ray Chiang added a comment - Keep forgetting to fix whitespace
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 40s Maven dependency ordering for branch
        +1 mvninstall 6m 46s trunk passed
        +1 compile 1m 29s trunk passed with JDK v1.8.0_72
        +1 compile 1m 42s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 24s trunk passed
        +1 mvnsite 1m 19s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 2m 21s trunk passed
        +1 javadoc 0m 55s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 3s the patch passed
        +1 compile 1m 26s the patch passed with JDK v1.8.0_72
        -1 javac 2m 39s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 360 unchanged - 0 fixed = 361 total (was 360)
        +1 javac 1m 26s the patch passed
        +1 compile 1m 41s the patch passed with JDK v1.7.0_95
        -1 javac 4m 20s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 365 unchanged - 0 fixed = 366 total (was 365)
        +1 javac 1m 41s the patch passed
        +1 checkstyle 0m 22s the patch passed
        +1 mvnsite 1m 13s the patch passed
        +1 mvneclipse 0m 32s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 50s the patch passed
        +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95
        +1 unit 1m 53s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 39s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        -1 unit 5m 21s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_72.
        +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        -1 unit 5m 42s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_95.
        -1 asflicense 0m 19s Patch generated 14 ASF License warnings.
        46m 49s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing
          hadoop.mapreduce.v2.hs.TestJobHistoryEvents
          hadoop.mapreduce.v2.hs.TestJobHistory
          hadoop.mapreduce.v2.hs.TestJobHistoryServer
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing
          hadoop.mapreduce.v2.hs.TestJobHistoryEvents
          hadoop.mapreduce.v2.hs.TestJobHistory
          hadoop.mapreduce.v2.hs.TestJobHistoryServer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789336/MAPREDUCE-6622.008.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 353d46c568ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4343a4c
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72.txt
        javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 40s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 1m 29s trunk passed with JDK v1.8.0_72 +1 compile 1m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 24s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 21s trunk passed +1 javadoc 0m 55s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 3s the patch passed +1 compile 1m 26s the patch passed with JDK v1.8.0_72 -1 javac 2m 39s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72 with JDK v1.8.0_72 generated 1 new + 360 unchanged - 0 fixed = 361 total (was 360) +1 javac 1m 26s the patch passed +1 compile 1m 41s the patch passed with JDK v1.7.0_95 -1 javac 4m 20s hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95 with JDK v1.7.0_95 generated 1 new + 365 unchanged - 0 fixed = 366 total (was 365) +1 javac 1m 41s the patch passed +1 checkstyle 0m 22s the patch passed +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 32s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 50s the patch passed +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95 +1 unit 1m 53s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. +1 unit 0m 39s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. -1 unit 5m 21s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_72. +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 45s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. -1 unit 5m 42s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 19s Patch generated 14 ASF License warnings. 46m 49s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing   hadoop.mapreduce.v2.hs.TestJobHistoryEvents   hadoop.mapreduce.v2.hs.TestJobHistory   hadoop.mapreduce.v2.hs.TestJobHistoryServer JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryParsing   hadoop.mapreduce.v2.hs.TestJobHistoryEvents   hadoop.mapreduce.v2.hs.TestJobHistory   hadoop.mapreduce.v2.hs.TestJobHistoryServer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789336/MAPREDUCE-6622.008.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 353d46c568ac 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4343a4c Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_72.txt javac hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6347/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        kasha Karthik Kambatla added a comment -

        Wouldn't the current patch log an error even if a value for the new config is not specified. We might want to do a conf.get() followed by a Interger.parseInt, and a Math.min check?

        Show
        kasha Karthik Kambatla added a comment - Wouldn't the current patch log an error even if a value for the new config is not specified. We might want to do a conf.get() followed by a Interger.parseInt, and a Math.min check?
        Hide
        rchiang Ray Chiang added a comment -
        • Fix up tasks cache initialization
        • Add unit tests for testing cache type and size
        Show
        rchiang Ray Chiang added a comment - Fix up tasks cache initialization Add unit tests for testing cache type and size
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 7m 37s trunk passed
        +1 compile 1m 41s trunk passed with JDK v1.8.0_72
        +1 compile 1m 48s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 41s trunk passed
        +1 findbugs 2m 36s trunk passed
        +1 javadoc 1m 1s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 16s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 8s the patch passed
        +1 compile 1m 35s the patch passed with JDK v1.8.0_72
        +1 javac 1m 35s the patch passed
        +1 compile 1m 48s the patch passed with JDK v1.7.0_95
        +1 javac 1m 48s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 1m 16s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 3s the patch passed
        +1 javadoc 0m 56s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95
        +1 unit 2m 0s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 40s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        -1 unit 5m 50s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_72.
        -1 unit 2m 22s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95.
        +1 unit 0m 50s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        -1 unit 6m 9s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_95.
        -1 asflicense 0m 23s Patch generated 14 ASF License warnings.
        50m 30s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryServer
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.tools.TestCLI
          hadoop.mapreduce.v2.hs.TestJobHistoryServer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789628/MAPREDUCE-6622.009.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux ca77615721d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d1dd248
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 7m 37s trunk passed +1 compile 1m 41s trunk passed with JDK v1.8.0_72 +1 compile 1m 48s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 2m 36s trunk passed +1 javadoc 1m 1s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 16s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 1m 35s the patch passed with JDK v1.8.0_72 +1 javac 1m 35s the patch passed +1 compile 1m 48s the patch passed with JDK v1.7.0_95 +1 javac 1m 48s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 16s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 3s the patch passed +1 javadoc 0m 56s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 5s the patch passed with JDK v1.7.0_95 +1 unit 2m 0s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. +1 unit 0m 40s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. -1 unit 5m 50s hadoop-mapreduce-client-hs in the patch failed with JDK v1.8.0_72. -1 unit 2m 22s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. +1 unit 0m 50s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. -1 unit 6m 9s hadoop-mapreduce-client-hs in the patch failed with JDK v1.7.0_95. -1 asflicense 0m 23s Patch generated 14 ASF License warnings. 50m 30s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.v2.hs.TestJobHistoryServer JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.tools.TestCLI   hadoop.mapreduce.v2.hs.TestJobHistoryServer Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789628/MAPREDUCE-6622.009.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ca77615721d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d1dd248 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6348/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -
        • Fixes that allow TestJobHistoryServer to pass.
          • Fix loadJob() so that it never returns null. Throw an exception if the history for the job cannot be found.
          • Fix getFullJob() so that the exception thrown in loadJob() is caught.
        Show
        rchiang Ray Chiang added a comment - Fixes that allow TestJobHistoryServer to pass. Fix loadJob() so that it never returns null. Throw an exception if the history for the job cannot be found. Fix getFullJob() so that the exception thrown in loadJob() is caught.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 6m 51s trunk passed
        +1 compile 1m 31s trunk passed with JDK v1.8.0_72
        +1 compile 1m 42s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 19s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 2m 21s trunk passed
        +1 javadoc 0m 54s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 4s the patch passed
        +1 compile 1m 28s the patch passed with JDK v1.8.0_72
        +1 javac 1m 28s the patch passed
        +1 compile 1m 41s the patch passed with JDK v1.7.0_95
        +1 javac 1m 41s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 1m 15s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 2m 48s the patch passed
        +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 0s the patch passed with JDK v1.7.0_95
        -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72.
        +1 unit 0m 41s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        +1 unit 5m 44s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72.
        -1 unit 2m 14s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95.
        +1 unit 0m 47s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        +1 unit 5m 59s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 19s Patch generated 14 ASF License warnings.
        47m 26s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.tools.TestCLI
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.tools.TestCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790004/MAPREDUCE-6622.010.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux f1bf84d2eee2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c4d4df8
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 51s trunk passed +1 compile 1m 31s trunk passed with JDK v1.8.0_72 +1 compile 1m 42s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 19s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 21s trunk passed +1 javadoc 0m 54s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 4s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 1m 28s the patch passed with JDK v1.8.0_72 +1 javac 1m 28s the patch passed +1 compile 1m 41s the patch passed with JDK v1.7.0_95 +1 javac 1m 41s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 1m 15s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 48s the patch passed +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 0s the patch passed with JDK v1.7.0_95 -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72. +1 unit 0m 41s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. +1 unit 5m 44s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72. -1 unit 2m 14s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. +1 unit 0m 47s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. +1 unit 5m 59s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 19s Patch generated 14 ASF License warnings. 47m 26s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.tools.TestCLI JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790004/MAPREDUCE-6622.010.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux f1bf84d2eee2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c4d4df8 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6350/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        RE: Failing unit test

        MAPREDUCE-6625 has been filed for testCLI#testGetJob()

        Show
        rchiang Ray Chiang added a comment - RE: Failing unit test MAPREDUCE-6625 has been filed for testCLI#testGetJob()
        Hide
        kasha Karthik Kambatla added a comment -

        Almost there. One comment:

        1. Should we use a more specific Exception in loadJob(). We don't distinguish between IOException due to failure to read (fileInfo.loadJob) and the null/deleted file. We should distinguish those so we can throw YanrRuntimeException if IOException.
        Show
        kasha Karthik Kambatla added a comment - Almost there. One comment: Should we use a more specific Exception in loadJob(). We don't distinguish between IOException due to failure to read (fileInfo.loadJob) and the null/deleted file. We should distinguish those so we can throw YanrRuntimeException if IOException.
        Hide
        rchiang Ray Chiang added a comment -
        • Change loadJob() to throw YarnRuntimeException instead of Exception
        • Fix try/catch to distinguish between different error cases
        Show
        rchiang Ray Chiang added a comment - Change loadJob() to throw YarnRuntimeException instead of Exception Fix try/catch to distinguish between different error cases
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 39s Maven dependency ordering for branch
        +1 mvninstall 7m 23s trunk passed
        +1 compile 1m 40s trunk passed with JDK v1.8.0_72
        +1 compile 1m 53s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 41s trunk passed
        +1 findbugs 2m 33s trunk passed
        +1 javadoc 0m 59s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 8s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 8s the patch passed
        +1 compile 1m 41s the patch passed with JDK v1.8.0_72
        +1 javac 1m 41s the patch passed
        +1 compile 1m 48s the patch passed with JDK v1.7.0_95
        +1 javac 1m 48s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 1m 20s the patch passed
        +1 mvneclipse 0m 34s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 7s the patch passed
        +1 javadoc 0m 53s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95
        -1 unit 2m 6s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72.
        +1 unit 0m 43s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        +1 unit 5m 55s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72.
        +1 unit 2m 25s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 50s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        +1 unit 6m 6s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 21s Patch generated 14 ASF License warnings.
        51m 3s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.tools.TestCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790037/MAPREDUCE-6622.011.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8b9c12ca4660 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d7fdec1
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 1m 40s trunk passed with JDK v1.8.0_72 +1 compile 1m 53s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 2m 33s trunk passed +1 javadoc 0m 59s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 8s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 1m 41s the patch passed with JDK v1.8.0_72 +1 javac 1m 41s the patch passed +1 compile 1m 48s the patch passed with JDK v1.7.0_95 +1 javac 1m 48s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 1m 20s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 7s the patch passed +1 javadoc 0m 53s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 4s the patch passed with JDK v1.7.0_95 -1 unit 2m 6s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_72. +1 unit 0m 43s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. +1 unit 5m 55s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72. +1 unit 2m 25s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 50s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. +1 unit 6m 6s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 21s Patch generated 14 ASF License warnings. 51m 3s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790037/MAPREDUCE-6622.011.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8b9c12ca4660 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d7fdec1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6351/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -

        No jenkins run. Relaunching.

        Show
        rchiang Ray Chiang added a comment - No jenkins run. Relaunching.
        Hide
        rchiang Ray Chiang added a comment -

        Re-launch test.

        Show
        rchiang Ray Chiang added a comment - Re-launch test.
        Hide
        rchiang Ray Chiang added a comment -

        Ugh, never mind. Had two windows opened and one not refreshed.

        Show
        rchiang Ray Chiang added a comment - Ugh, never mind. Had two windows opened and one not refreshed.
        Hide
        rchiang Ray Chiang added a comment -
        • Add unit test to verify getFullJob() return value with exceptions is backwards compatible
        • Fix loadJob() and getFullJob() to catch/pass certain exceptions
        Show
        rchiang Ray Chiang added a comment - Add unit test to verify getFullJob() return value with exceptions is backwards compatible Fix loadJob() and getFullJob() to catch/pass certain exceptions
        Hide
        kasha Karthik Kambatla added a comment -

        The latest patch does fix the behavior, nice.

        Minor nits:

        1. I suppose the reason we are setting the message for HSFileException is to log it. However, the code doesn't log anything. We should do one of the two:
          1. Log before throwing HSFileException and don't set the message.
          2. Log when we catch HSFileException.
        2. The test uses wildcard imports - let us avoid that.
        Show
        kasha Karthik Kambatla added a comment - The latest patch does fix the behavior, nice. Minor nits: I suppose the reason we are setting the message for HSFileException is to log it. However, the code doesn't log anything. We should do one of the two: Log before throwing HSFileException and don't set the message. Log when we catch HSFileException. The test uses wildcard imports - let us avoid that.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 2m 19s Docker failed to build yetus/hadoop:0ca8df7.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790250/MAPREDUCE-6622.012.patch
        JIRA Issue MAPREDUCE-6622
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6352/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 2m 19s Docker failed to build yetus/hadoop:0ca8df7. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790250/MAPREDUCE-6622.012.patch JIRA Issue MAPREDUCE-6622 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6352/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rchiang Ray Chiang added a comment -
        • Remove wildcard imports
        • Add logging call when HSFileRuntimeException is caught
        Show
        rchiang Ray Chiang added a comment - Remove wildcard imports Add logging call when HSFileRuntimeException is caught
        Hide
        kasha Karthik Kambatla added a comment -

        Latest patch looks good to me. +1. Thanks for your patience through the reviews, Ray Chiang.

        Show
        kasha Karthik Kambatla added a comment - Latest patch looks good to me. +1. Thanks for your patience through the reviews, Ray Chiang .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 12m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 18s Maven dependency ordering for branch
        +1 mvninstall 6m 38s trunk passed
        +1 compile 1m 26s trunk passed with JDK v1.8.0_72
        +1 compile 1m 40s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 23s trunk passed
        +1 mvnsite 1m 17s trunk passed
        +1 mvneclipse 0m 38s trunk passed
        +1 findbugs 2m 23s trunk passed
        +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 4s the patch passed
        +1 compile 1m 25s the patch passed with JDK v1.8.0_72
        +1 javac 1m 25s the patch passed
        +1 compile 1m 38s the patch passed with JDK v1.7.0_95
        +1 javac 1m 38s the patch passed
        -1 checkstyle 0m 21s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47)
        +1 mvnsite 1m 13s the patch passed
        +1 mvneclipse 0m 33s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        -1 findbugs 0m 44s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
        +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72
        +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95
        +1 unit 1m 52s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 38s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72.
        +1 unit 5m 33s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72.
        +1 unit 2m 14s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 unit 0m 46s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95.
        +1 unit 5m 55s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95.
        -1 asflicense 0m 20s Patch generated 14 ASF License warnings.
        58m 42s



        Reason Tests
        FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
          Should org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$HSFileRuntimeException be a static inner class? At CachedHistoryStorage.java:inner class? At CachedHistoryStorage.java:[line 163]



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790252/MAPREDUCE-6622.013.patch
        JIRA Issue MAPREDUCE-6622
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 4b01c25ee2f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d1d4e16
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.html
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 12m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 1m 26s trunk passed with JDK v1.8.0_72 +1 compile 1m 40s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 23s trunk passed +1 mvnsite 1m 17s trunk passed +1 mvneclipse 0m 38s trunk passed +1 findbugs 2m 23s trunk passed +1 javadoc 0m 52s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 5s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 4s the patch passed +1 compile 1m 25s the patch passed with JDK v1.8.0_72 +1 javac 1m 25s the patch passed +1 compile 1m 38s the patch passed with JDK v1.7.0_95 +1 javac 1m 38s the patch passed -1 checkstyle 0m 21s hadoop-mapreduce-project/hadoop-mapreduce-client: patch generated 1 new + 47 unchanged - 0 fixed = 48 total (was 47) +1 mvnsite 1m 13s the patch passed +1 mvneclipse 0m 33s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. -1 findbugs 0m 44s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 49s the patch passed with JDK v1.8.0_72 +1 javadoc 0m 59s the patch passed with JDK v1.7.0_95 +1 unit 1m 52s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_72. +1 unit 0m 38s hadoop-mapreduce-client-common in the patch passed with JDK v1.8.0_72. +1 unit 5m 33s hadoop-mapreduce-client-hs in the patch passed with JDK v1.8.0_72. +1 unit 2m 14s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 unit 0m 46s hadoop-mapreduce-client-common in the patch passed with JDK v1.7.0_95. +1 unit 5m 55s hadoop-mapreduce-client-hs in the patch passed with JDK v1.7.0_95. -1 asflicense 0m 20s Patch generated 14 ASF License warnings. 58m 42s Reason Tests FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs   Should org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage$HSFileRuntimeException be a static inner class? At CachedHistoryStorage.java:inner class? At CachedHistoryStorage.java: [line 163] Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790252/MAPREDUCE-6622.013.patch JIRA Issue MAPREDUCE-6622 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4b01c25ee2f0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d1d4e16 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-hs.html JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U: hadoop-mapreduce-project/hadoop-mapreduce-client Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6353/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        rkanter Robert Kanter added a comment -

        +1

        Will commit this shortly. The findbugs and checkstyle are very trivial. I'll just fix those during commit so we don't have to go through another iteration for just them.

        Show
        rkanter Robert Kanter added a comment - +1 Will commit this shortly. The findbugs and checkstyle are very trivial. I'll just fix those during commit so we don't have to go through another iteration for just them.
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Ray Chiang and everyone for reviews. Committed to trunk and branch-2!

        Show
        rkanter Robert Kanter added a comment - Thanks Ray Chiang and everyone for reviews. Committed to trunk and branch-2!
        Hide
        rkanter Robert Kanter added a comment -

        For completeness, patch 14 is the final commit with the trivial changes I made.

        Show
        rkanter Robert Kanter added a comment - For completeness, patch 14 is the final commit with the trivial changes I made.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9380 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9380/)
        MAPREDUCE-6622. Add capability to set JHS job cache to a task-based (rkanter: rev 0f72da7e281376f4fcbfbf3fb33f5d7fedcdb1aa)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistory.java
        • hadoop-mapreduce-project/CHANGES.txt
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java
        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9380 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9380/ ) MAPREDUCE-6622 . Add capability to set JHS job cache to a task-based (rkanter: rev 0f72da7e281376f4fcbfbf3fb33f5d7fedcdb1aa) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JHAdminConfig.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistory.java hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
        Hide
        zxu zhihai xu added a comment -

        This patch also fixed a memory leak issue due to a race condition at CachedHistoryStorage.getFullJob, We can reproduce this memory leak issue by keeping refreshing the JHS web page for a job with more than 40,000 mappers quickly. The race condition is fileInfo.loadJob() takes long time to load the job with more than 40000 mappers, during that time, fileInfo.loadJob() is called multiple times for the same job because no synchronization between loadedJobCache.get(jobId) and loadJob(fileInfo). You will see the used heap memory quickly go up. Looked at the heap dump, we find 56 CompletedJob instances for the same job ID, which have total more 2 million mappers(56*40000). Based on the link from http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheBuilder.html#build(com.google.common.cache.CacheLoader)
        This won't be an issue for com.google.common.cache.LoadingCache:

        If another thread is currently loading the value for this key, simply waits for that thread to finish and returns its loaded value
        

        This looks like a critical issue for me. Should we backport this patch to 2.7.3 and 2.6.5 branch?

        Show
        zxu zhihai xu added a comment - This patch also fixed a memory leak issue due to a race condition at CachedHistoryStorage.getFullJob , We can reproduce this memory leak issue by keeping refreshing the JHS web page for a job with more than 40,000 mappers quickly. The race condition is fileInfo.loadJob() takes long time to load the job with more than 40000 mappers, during that time, fileInfo.loadJob() is called multiple times for the same job because no synchronization between loadedJobCache.get(jobId) and loadJob(fileInfo) . You will see the used heap memory quickly go up. Looked at the heap dump, we find 56 CompletedJob instances for the same job ID, which have total more 2 million mappers(56*40000). Based on the link from http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/cache/CacheBuilder.html#build(com.google.common.cache.CacheLoader ) This won't be an issue for com.google.common.cache.LoadingCache: If another thread is currently loading the value for this key, simply waits for that thread to finish and returns its loaded value This looks like a critical issue for me. Should we backport this patch to 2.7.3 and 2.6.5 branch?
        Hide
        rchiang Ray Chiang added a comment -

        Good observation, Zhihai.

        Since it's an optional setting that's off by default, I'd be fine with adding it to the 2.6/2.7 line.

        Show
        rchiang Ray Chiang added a comment - Good observation, Zhihai. Since it's an optional setting that's off by default, I'd be fine with adding it to the 2.6/2.7 line.
        Hide
        zxu zhihai xu added a comment -

        Thanks for the confirmation Ray Chiang! I will back port the patch to 2.6.5 and 2.7.3 branch after waiting for several days if no one objects.

        Show
        zxu zhihai xu added a comment - Thanks for the confirmation Ray Chiang ! I will back port the patch to 2.6.5 and 2.7.3 branch after waiting for several days if no one objects.
        Hide
        zxu zhihai xu added a comment -

        Committed it to both branch 2.6 and 2.7.

        Show
        zxu zhihai xu added a comment - Committed it to both branch 2.6 and 2.7.
        Hide
        zxu zhihai xu added a comment -

        Committed it to 2.8 also.

        Show
        zxu zhihai xu added a comment - Committed it to 2.8 also.
        Hide
        rchiang Ray Chiang added a comment -

        Thanks Zhihai!

        Show
        rchiang Ray Chiang added a comment - Thanks Zhihai!
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

          People

          • Assignee:
            rchiang Ray Chiang
            Reporter:
            rchiang Ray Chiang
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development