Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6.4
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There is a potential memory leak in CryptoOutputStream.java. It allocates two direct byte buffers (inBuffer and outBuffer) that get freed when close() method is called. Most of the time, close() method is called. However, when writing to intermediate Map output file or the spill files in MapTask, close() is never called since calling so would close the underlying stream which is not desirable. There is a single underlying physical stream that contains multiple logical streams one per partition of Map output.

      By default the amount of memory allocated per byte buffer is 128 KB and so the total memory allocated is 256 KB, This may not sound much. However, if the number of partitions (or number of reducers) is large (in the hundreds) and/or there are spill files created in MapTask, this can grow into a few hundred MB.

      I can think of two ways to address this issue:

      Possible Fix - 1

      According to JDK documentation:

      The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious. It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers that are subject to the underlying system's native I/O operations. In general it is best to allocate direct buffers only when they yield a measureable gain in program performance.

      It is not clear to me whether there is any benefit of allocating direct byte buffers in CryptoOutputStream.java. In fact, there is a slight CPU overhead in moving data from outBuffer to a temporary byte array as per the following code in CryptoOutputStream.java.

          /*
           * If underlying stream supports {@link ByteBuffer} write in future, needs
           * refine here. 
           */
          final byte[] tmp = getTmpBuf();
          outBuffer.get(tmp, 0, len);
          out.write(tmp, 0, len);
      

      Even if the underlying stream supports direct byte buffer IO (or direct IO in OS parlance), it is not clear whether it will yield any measurable performance gain.

      The fix would be to allocate a ByteBuffer on the heap for inBuffer and wrap a byte array in a ByteBuffer for outBuffer. By the way, the inBuffer and outBuffer have to be ByteBuffer as demanded by the encrypt() method in Encryptor.

      Possible Fix - 2

      Assuming that we want to keep the buffers as direct byte buffers, we can create a new constructor to CryptoOutputStream and pass a boolean flag ownOutputStream to indicate whether the underlying stream will be owned by CryptoOutputStream. If it is true, then calling the close() method will close the underlying stream. Otherwise, when close() is called only the direct byte buffers will be freed and the underlying stream will not be closed.

      The scope of changes for this fix will be somewhat wider. We need to modify MapTask.java, CryptoUtils.java, and CryptoFSDataOutputStream.java as well to pass the ownership flag mentioned above.

      I can post a patch for either of the above. I welcome any other ideas from developers to fix this issue.

      1. MAPREDUCE-6628.001.patch
        3 kB
        Mariappan Asokan
      2. MAPREDUCE-6628.002.patch
        12 kB
        Mariappan Asokan
      3. MAPREDUCE-6628.003.patch
        15 kB
        Mariappan Asokan
      4. MAPREDUCE-6628.004.patch
        26 kB
        Mariappan Asokan
      5. MAPREDUCE-6628.005.patch
        26 kB
        Mariappan Asokan
      6. MAPREDUCE-6628.006.patch
        26 kB
        Mariappan Asokan
      7. MAPREDUCE-6628.007.patch
        25 kB
        Mariappan Asokan
      8. MAPREDUCE-6628.008.patch
        17 kB
        Mariappan Asokan
      9. MAPREDUCE-6628.009.patch
        17 kB
        Mariappan Asokan

        Activity

        Hide
        andrew.wang Andrew Wang added a comment -

        Yi Liu, any thoughts on this one?

        Show
        andrew.wang Andrew Wang added a comment - Yi Liu , any thoughts on this one?
        Hide
        masokan Mariappan Asokan added a comment -

        Hi Andrew,
        I uploaded a patch based on "Possible Fix - 1." Can you please review it once Jenkins gives the green light?
        Thanks.

        Show
        masokan Mariappan Asokan added a comment - Hi Andrew, I uploaded a patch based on "Possible Fix - 1." Can you please review it once Jenkins gives the green light? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 47s trunk passed
        +1 compile 5m 56s trunk passed with JDK v1.8.0_77
        +1 compile 7m 5s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 34s trunk passed
        +1 javadoc 0m 57s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 42s the patch passed
        +1 compile 7m 10s the patch passed with JDK v1.8.0_77
        +1 javac 7m 10s the patch passed
        +1 compile 7m 22s the patch passed with JDK v1.7.0_95
        +1 javac 7m 22s the patch passed
        +1 checkstyle 0m 21s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 51s the patch passed
        +1 javadoc 0m 57s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95
        -1 unit 21m 37s hadoop-common in the patch failed with JDK v1.8.0_77.
        -1 unit 6m 44s hadoop-common in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        75m 46s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
          hadoop.crypto.TestCryptoStreamsNormal
          hadoop.crypto.TestCryptoCodec
          hadoop.crypto.TestCryptoStreams
          hadoop.crypto.TestCryptoStreamsForLocalFS
        JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
        JDK v1.7.0_95 Failed junit tests hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec
          hadoop.crypto.TestCryptoStreamsNormal
          hadoop.crypto.TestCryptoCodec
          hadoop.crypto.TestCryptoStreams
          hadoop.crypto.TestCryptoStreamsForLocalFS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797560/MAPREDUCE-6628.001.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 7539623d0fb7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a62637a
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 47s trunk passed +1 compile 5m 56s trunk passed with JDK v1.8.0_77 +1 compile 7m 5s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 34s trunk passed +1 javadoc 0m 57s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 3s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 42s the patch passed +1 compile 7m 10s the patch passed with JDK v1.8.0_77 +1 javac 7m 10s the patch passed +1 compile 7m 22s the patch passed with JDK v1.7.0_95 +1 javac 7m 22s the patch passed +1 checkstyle 0m 21s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 57s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 6s the patch passed with JDK v1.7.0_95 -1 unit 21m 37s hadoop-common in the patch failed with JDK v1.8.0_77. -1 unit 6m 44s hadoop-common in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 75m 46s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec   hadoop.crypto.TestCryptoStreamsNormal   hadoop.crypto.TestCryptoCodec   hadoop.crypto.TestCryptoStreams   hadoop.crypto.TestCryptoStreamsForLocalFS JDK v1.8.0_77 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_95 Failed junit tests hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec   hadoop.crypto.TestCryptoStreamsNormal   hadoop.crypto.TestCryptoCodec   hadoop.crypto.TestCryptoStreams   hadoop.crypto.TestCryptoStreamsForLocalFS Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797560/MAPREDUCE-6628.001.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 7539623d0fb7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a62637a Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_77.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6419/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        Looked at the test failures. The JNI implementation for OpenSSLCipher requires direct buffers perhaps to avoid excessive data moves across Java and C. So I will create a patch based on "Possible Fix - 2" which will retain the direct buffers.

        Show
        masokan Mariappan Asokan added a comment - Looked at the test failures. The JNI implementation for OpenSSLCipher requires direct buffers perhaps to avoid excessive data moves across Java and C. So I will create a patch based on "Possible Fix - 2" which will retain the direct buffers.
        Hide
        masokan Mariappan Asokan added a comment -

        Uploaded a patch based on "Possible Fix - 2."

        Show
        masokan Mariappan Asokan added a comment - Uploaded a patch based on "Possible Fix - 2."
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 9s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 7m 6s trunk passed
        +1 compile 6m 42s trunk passed with JDK v1.8.0_77
        +1 compile 6m 50s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 7s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 2m 37s trunk passed
        +1 javadoc 1m 18s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 33s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 5s the patch passed
        +1 compile 5m 45s the patch passed with JDK v1.8.0_77
        +1 javac 5m 45s the patch passed
        +1 compile 6m 40s the patch passed with JDK v1.7.0_95
        +1 javac 6m 40s the patch passed
        -1 checkstyle 1m 5s root: patch generated 1 new + 232 unchanged - 4 fixed = 233 total (was 236)
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 3m 1s the patch passed
        -1 javadoc 2m 45s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77 with JDK v1.8.0_77 generated 4 new + 96 unchanged - 4 fixed = 100 total (was 100)
        +1 javadoc 1m 12s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 29s the patch passed with JDK v1.7.0_95
        +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.8.0_77.
        +1 unit 1m 57s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_77.
        +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 22s Patch does not generate ASF License warnings.
        73m 22s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797807/MAPREDUCE-6628.002.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b8324bd2b8b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / ec06957
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/diff-checkstyle-root.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/whitespace-eol.txt
        javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 7m 6s trunk passed +1 compile 6m 42s trunk passed with JDK v1.8.0_77 +1 compile 6m 50s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 7s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 37s trunk passed +1 javadoc 1m 18s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 33s trunk passed with JDK v1.7.0_95 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 5m 45s the patch passed with JDK v1.8.0_77 +1 javac 5m 45s the patch passed +1 compile 6m 40s the patch passed with JDK v1.7.0_95 +1 javac 6m 40s the patch passed -1 checkstyle 1m 5s root: patch generated 1 new + 232 unchanged - 4 fixed = 233 total (was 236) +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 28s the patch passed -1 whitespace 0m 0s The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 1s the patch passed -1 javadoc 2m 45s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77 with JDK v1.8.0_77 generated 4 new + 96 unchanged - 4 fixed = 100 total (was 100) +1 javadoc 1m 12s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 29s the patch passed with JDK v1.7.0_95 +1 unit 7m 33s hadoop-common in the patch passed with JDK v1.8.0_77. +1 unit 1m 57s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_77. +1 unit 7m 58s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 22s Patch does not generate ASF License warnings. 73m 22s Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12797807/MAPREDUCE-6628.002.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b8324bd2b8b6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ec06957 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/diff-checkstyle-root.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/whitespace-eol.txt javadoc hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/artifact/patchprocess/diff-javadoc-javadoc-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6424/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 28s trunk passed
        +1 compile 5m 43s trunk passed with JDK v1.8.0_77
        +1 compile 6m 36s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 6s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 2m 33s trunk passed
        +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77
        +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 5s the patch passed
        +1 compile 5m 36s the patch passed with JDK v1.8.0_77
        +1 javac 5m 36s the patch passed
        +1 compile 6m 36s the patch passed with JDK v1.7.0_95
        +1 javac 6m 36s the patch passed
        +1 checkstyle 1m 8s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236)
        +1 mvnsite 1m 27s the patch passed
        +1 mvneclipse 0m 29s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 4s the patch passed
        +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77
        +1 javadoc 1m 33s the patch passed with JDK v1.7.0_95
        +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.8.0_77.
        -1 unit 2m 1s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_77.
        +1 unit 7m 54s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 29s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        71m 50s



        Reason Tests
        JDK v1.8.0_77 Failed junit tests hadoop.mapreduce.tools.TestCLI



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798032/MAPREDUCE-6628.003.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 879e99ccd2cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1ff27f9
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 28s trunk passed +1 compile 5m 43s trunk passed with JDK v1.8.0_77 +1 compile 6m 36s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 33s trunk passed +1 javadoc 1m 14s trunk passed with JDK v1.8.0_77 +1 javadoc 1m 28s trunk passed with JDK v1.7.0_95 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 5s the patch passed +1 compile 5m 36s the patch passed with JDK v1.8.0_77 +1 javac 5m 36s the patch passed +1 compile 6m 36s the patch passed with JDK v1.7.0_95 +1 javac 6m 36s the patch passed +1 checkstyle 1m 8s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236) +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 29s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 4s the patch passed +1 javadoc 1m 19s the patch passed with JDK v1.8.0_77 +1 javadoc 1m 33s the patch passed with JDK v1.7.0_95 +1 unit 7m 38s hadoop-common in the patch passed with JDK v1.8.0_77. -1 unit 2m 1s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_77. +1 unit 7m 54s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 2m 29s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 71m 50s Reason Tests JDK v1.8.0_77 Failed junit tests hadoop.mapreduce.tools.TestCLI Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12798032/MAPREDUCE-6628.003.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 879e99ccd2cb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1ff27f9 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_77 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_77.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6426/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        I created a test which prints out memory statistics (works only on Linux) before and after the sort phase in the map task. I specified an encryption buffer size of 64 MB. I did this to keep the number of reducers small so that the test would finish fast. I could have left the buffer size as default (128 KB) and increased the number of reducers to more than 5000 to get the same effect. In any case, following are the statistics before the patch.

        Memory statistics before the patch

        MapOutputCopier.init(): Memory usage:
        VmPeak: 1326992 kB
        VmSize: 1326992 kB
        VmLck: 0 kB
        VmPin: 0 kB
        VmHWM: 228188 kB
        VmRSS: 228188 kB
        VmData: 1256456 kB
        VmStk: 156 kB
        VmExe: 4 kB
        VmLib: 15908 kB
        VmPTE: 676 kB
        VmSwap: 0 kB
        MapOutputCopier.close(): Memory usage:
        VmPeak: 2049988 kB
        VmSize: 1722288 kB
        VmLck: 0 kB
        VmPin: 0 kB
        VmHWM: 1335640 kB
        VmRSS: 1069044 kB
        VmData: 1651752 kB
        VmStk: 156 kB
        VmExe: 4 kB
        VmLib: 15908 kB
        VmPTE: 2352 kB
        VmSwap: 0 kB

        After the patch was applied, I reran the test and here are the statistics:

        Memory statistics after the patch

        MapOutputCopier.init(): Memory usage:
        VmPeak: 1325348 kB
        VmSize: 1325348 kB
        VmLck: 0 kB
        VmPin: 0 kB
        VmHWM: 228308 kB
        VmRSS: 228308 kB
        VmData: 1254812 kB
        VmStk: 156 kB
        VmExe: 4 kB
        VmLib: 15908 kB
        VmPTE: 676 kB
        VmSwap: 0 kB
        MapOutputCopier.close(): Memory usage:
        VmPeak: 1460344 kB
        VmSize: 1329264 kB
        VmLck: 0 kB
        VmPin: 0 kB
        VmHWM: 922732 kB
        VmRSS: 795800 kB
        VmData: 1258728 kB
        VmStk: 156 kB
        VmExe: 4 kB
        VmLib: 15908 kB
        VmPTE: 1800 kB
        VmSwap: 0 kB

        It can be noticed that after the sort completes, the physical as well virtual memory usage are much higher in the case before the patch.

        I am including the test as part of the patch. However, the test itself cannot verify on each run that memory was freed properly since there is no specific value to check against. The memory statistics are output in stdout of the map task.

        Show
        masokan Mariappan Asokan added a comment - I created a test which prints out memory statistics (works only on Linux) before and after the sort phase in the map task. I specified an encryption buffer size of 64 MB. I did this to keep the number of reducers small so that the test would finish fast. I could have left the buffer size as default (128 KB) and increased the number of reducers to more than 5000 to get the same effect. In any case, following are the statistics before the patch. Memory statistics before the patch MapOutputCopier.init(): Memory usage: VmPeak: 1326992 kB VmSize: 1326992 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 228188 kB VmRSS: 228188 kB VmData: 1256456 kB VmStk: 156 kB VmExe: 4 kB VmLib: 15908 kB VmPTE: 676 kB VmSwap: 0 kB MapOutputCopier.close(): Memory usage: VmPeak: 2049988 kB VmSize: 1722288 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 1335640 kB VmRSS: 1069044 kB VmData: 1651752 kB VmStk: 156 kB VmExe: 4 kB VmLib: 15908 kB VmPTE: 2352 kB VmSwap: 0 kB After the patch was applied, I reran the test and here are the statistics: Memory statistics after the patch MapOutputCopier.init(): Memory usage: VmPeak: 1325348 kB VmSize: 1325348 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 228308 kB VmRSS: 228308 kB VmData: 1254812 kB VmStk: 156 kB VmExe: 4 kB VmLib: 15908 kB VmPTE: 676 kB VmSwap: 0 kB MapOutputCopier.close(): Memory usage: VmPeak: 1460344 kB VmSize: 1329264 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 922732 kB VmRSS: 795800 kB VmData: 1258728 kB VmStk: 156 kB VmExe: 4 kB VmLib: 15908 kB VmPTE: 1800 kB VmSwap: 0 kB It can be noticed that after the sort completes, the physical as well virtual memory usage are much higher in the case before the patch. I am including the test as part of the patch. However, the test itself cannot verify on each run that memory was freed properly since there is no specific value to check against. The memory statistics are output in stdout of the map task.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 10m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 39s trunk passed
        +1 compile 5m 41s trunk passed with JDK v1.8.0_92
        +1 compile 6m 32s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 5s trunk passed
        +1 mvnsite 1m 52s trunk passed
        +1 mvneclipse 0m 40s trunk passed
        +1 findbugs 2m 59s trunk passed
        +1 javadoc 1m 23s trunk passed with JDK v1.8.0_92
        +1 javadoc 1m 40s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 26s the patch passed
        +1 compile 5m 44s the patch passed with JDK v1.8.0_92
        +1 javac 5m 44s the patch passed
        +1 compile 6m 39s the patch passed with JDK v1.7.0_95
        +1 javac 6m 39s the patch passed
        +1 checkstyle 1m 6s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236)
        +1 mvnsite 1m 52s the patch passed
        +1 mvneclipse 0m 38s the patch passed
        -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 findbugs 3m 39s the patch passed
        +1 javadoc 1m 24s the patch passed with JDK v1.8.0_92
        +1 javadoc 1m 40s the patch passed with JDK v1.7.0_95
        -1 unit 16m 52s hadoop-common in the patch failed with JDK v1.8.0_92.
        +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_92.
        -1 unit 105m 26s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_92.
        +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 18s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        -1 unit 104m 16s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 33s Patch does not generate ASF License warnings.
        305m 33s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.mapreduce.v2.TestUberAM
        JDK v1.8.0_92 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:fbe3e86
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800622/MAPREDUCE-6628.004.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b2d1026c965c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f308561
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/whitespace-eol.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 10m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 39s trunk passed +1 compile 5m 41s trunk passed with JDK v1.8.0_92 +1 compile 6m 32s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 5s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 2m 59s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_92 +1 javadoc 1m 40s trunk passed with JDK v1.7.0_95 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 26s the patch passed +1 compile 5m 44s the patch passed with JDK v1.8.0_92 +1 javac 5m 44s the patch passed +1 compile 6m 39s the patch passed with JDK v1.7.0_95 +1 javac 6m 39s the patch passed +1 checkstyle 1m 6s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236) +1 mvnsite 1m 52s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 3m 39s the patch passed +1 javadoc 1m 24s the patch passed with JDK v1.8.0_92 +1 javadoc 1m 40s the patch passed with JDK v1.7.0_95 -1 unit 16m 52s hadoop-common in the patch failed with JDK v1.8.0_92. +1 unit 1m 54s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_92. -1 unit 105m 26s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_92. +1 unit 8m 20s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 2m 18s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. -1 unit 104m 16s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 33s Patch does not generate ASF License warnings. 305m 33s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.mapreduce.v2.TestUberAM JDK v1.8.0_92 Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Image:yetus/hadoop:fbe3e86 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12800622/MAPREDUCE-6628.004.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b2d1026c965c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f308561 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6456/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        Got rid of trailing spaces in the test and uploaded a new patch. The failing tests seem to be unrelated to this patch.

        Show
        masokan Mariappan Asokan added a comment - Got rid of trailing spaces in the test and uploaded a new patch. The failing tests seem to be unrelated to this patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 6m 38s trunk passed
        +1 compile 6m 14s trunk passed with JDK v1.8.0_92
        +1 compile 6m 39s trunk passed with JDK v1.7.0_95
        +1 checkstyle 1m 6s trunk passed
        +1 mvnsite 1m 57s trunk passed
        +1 mvneclipse 0m 39s trunk passed
        +1 findbugs 3m 6s trunk passed
        +1 javadoc 1m 34s trunk passed with JDK v1.8.0_92
        +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 29s the patch passed
        +1 compile 6m 26s the patch passed with JDK v1.8.0_92
        +1 javac 6m 26s the patch passed
        +1 compile 6m 48s the patch passed with JDK v1.7.0_95
        +1 javac 6m 48s the patch passed
        +1 checkstyle 1m 8s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236)
        +1 mvnsite 1m 54s the patch passed
        +1 mvneclipse 0m 41s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 3m 44s the patch passed
        +1 javadoc 1m 22s the patch passed with JDK v1.8.0_92
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95
        -1 unit 7m 22s hadoop-common in the patch failed with JDK v1.8.0_92.
        -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_92.
        -1 unit 108m 26s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_92.
        +1 unit 8m 7s hadoop-common in the patch passed with JDK v1.7.0_95.
        +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95.
        -1 unit 108m 0s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 33s Patch does not generate ASF License warnings.
        293m 52s



        Reason Tests
        JDK v1.8.0_92 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          hadoop.mapreduce.tools.TestCLI
          hadoop.mapreduce.v2.TestUberAM
        JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801306/MAPREDUCE-6628.005.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 69a21900f0bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6243eab
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 6m 14s trunk passed with JDK v1.8.0_92 +1 compile 6m 39s trunk passed with JDK v1.7.0_95 +1 checkstyle 1m 6s trunk passed +1 mvnsite 1m 57s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 3m 6s trunk passed +1 javadoc 1m 34s trunk passed with JDK v1.8.0_92 +1 javadoc 1m 43s trunk passed with JDK v1.7.0_95 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 6m 26s the patch passed with JDK v1.8.0_92 +1 javac 6m 26s the patch passed +1 compile 6m 48s the patch passed with JDK v1.7.0_95 +1 javac 6m 48s the patch passed +1 checkstyle 1m 8s root: patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236) +1 mvnsite 1m 54s the patch passed +1 mvneclipse 0m 41s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 3m 44s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_92 +1 javadoc 1m 42s the patch passed with JDK v1.7.0_95 -1 unit 7m 22s hadoop-common in the patch failed with JDK v1.8.0_92. -1 unit 1m 55s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_92. -1 unit 108m 26s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_92. +1 unit 8m 7s hadoop-common in the patch passed with JDK v1.7.0_95. +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_95. -1 unit 108m 0s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 33s Patch does not generate ASF License warnings. 293m 52s Reason Tests JDK v1.8.0_92 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.mapreduce.tools.TestCLI   hadoop.mapreduce.v2.TestUberAM JDK v1.7.0_95 Failed junit tests hadoop.mapreduce.v2.TestUberAM Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12801306/MAPREDUCE-6628.005.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 69a21900f0bb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6243eab Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_92 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_92.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6468/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        Hi Andrew,
        The test failure seems to be unrelated to this patch. Can you please review the patch and commit it? Thanks.

        Show
        masokan Mariappan Asokan added a comment - Hi Andrew, The test failure seems to be unrelated to this patch. Can you please review the patch and commit it? Thanks.
        Hide
        andrew.wang Andrew Wang added a comment -

        Thanks for sticking with this for so long Mariappan. The stream-related changes overall look good to me. One naming nit, could we call the boolean "closeWrapperStream" rather than "ownOutputStream"? I think that's more descriptive.

        The test should also be JUnit4 rather than JUnit3.

        Can someone more familiar with the MR side review the MapTask and unit test changes? It'd also be good to get confirmation about the overall idea from an MR person.

        Show
        andrew.wang Andrew Wang added a comment - Thanks for sticking with this for so long Mariappan. The stream-related changes overall look good to me. One naming nit, could we call the boolean "closeWrapperStream" rather than "ownOutputStream"? I think that's more descriptive. The test should also be JUnit4 rather than JUnit3. Can someone more familiar with the MR side review the MapTask and unit test changes? It'd also be good to get confirmation about the overall idea from an MR person.
        Hide
        masokan Mariappan Asokan added a comment -

        Andrew,
        Thanks for looking at the patch. Sure, I will change the name of the parameter and make the new test JUnit4 compliant. I will wait for anyone else to comment on the MR side and upload a new patch.

        Show
        masokan Mariappan Asokan added a comment - Andrew, Thanks for looking at the patch. Sure, I will change the name of the parameter and make the new test JUnit4 compliant. I will wait for anyone else to comment on the MR side and upload a new patch.
        Hide
        masokan Mariappan Asokan added a comment -

        Andrew,
        I changed the name of the parameter and made the new test Juni4 compliant. Any MR persons to comment on the MR side changes? Thanks.

        Show
        masokan Mariappan Asokan added a comment - Andrew, I changed the name of the parameter and made the new test Juni4 compliant. Any MR persons to comment on the MR side changes? Thanks.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 11s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 8s Maven dependency ordering for branch
        -1 mvninstall 0m 8s root in trunk failed.
        +1 compile 7m 19s trunk passed with JDK v1.8.0_91
        -1 compile 0m 9s root in trunk failed with JDK v1.7.0_95.
        +1 checkstyle 1m 56s trunk passed
        -1 mvnsite 0m 16s hadoop-common in trunk failed.
        -1 mvnsite 0m 14s hadoop-mapreduce-client-core in trunk failed.
        -1 mvnsite 0m 14s hadoop-mapreduce-client-jobclient in trunk failed.
        -1 mvneclipse 0m 8s hadoop-common in trunk failed.
        -1 findbugs 0m 8s hadoop-common in trunk failed.
        -1 findbugs 0m 11s hadoop-mapreduce-client-core in trunk failed.
        -1 findbugs 0m 11s hadoop-mapreduce-client-jobclient in trunk failed.
        -1 javadoc 0m 7s hadoop-common in trunk failed with JDK v1.8.0_91.
        -1 javadoc 0m 9s hadoop-common in trunk failed with JDK v1.7.0_95.
        0 mvndep 0m 9s Maven dependency ordering for patch
        -1 mvninstall 0m 8s hadoop-common in the patch failed.
        -1 mvninstall 0m 12s hadoop-mapreduce-client-core in the patch failed.
        -1 mvninstall 0m 11s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 compile 6m 48s the patch passed with JDK v1.8.0_91
        +1 javac 6m 48s the patch passed
        -1 compile 0m 10s root in the patch failed with JDK v1.7.0_95.
        -1 javac 0m 10s root in the patch failed with JDK v1.7.0_95.
        -1 checkstyle 1m 26s root: patch generated 32 new + 232 unchanged - 4 fixed = 264 total (was 236)
        -1 mvnsite 0m 12s hadoop-common in the patch failed.
        -1 mvnsite 0m 14s hadoop-mapreduce-client-core in the patch failed.
        -1 mvnsite 0m 13s hadoop-mapreduce-client-jobclient in the patch failed.
        -1 mvneclipse 0m 8s hadoop-common in the patch failed.
        -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 findbugs 0m 8s hadoop-common in the patch failed.
        -1 findbugs 0m 12s hadoop-mapreduce-client-core in the patch failed.
        -1 findbugs 0m 10s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 javadoc 1m 29s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91 with JDK v1.8.0_91 generated 0 new + 2497 unchanged - 12 fixed = 2497 total (was 2509)
        -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91.
        +1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91.
        +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.8.0_91.
        -1 javadoc 0m 9s hadoop-common in the patch failed with JDK v1.7.0_95.
        -1 unit 0m 7s hadoop-common in the patch failed with JDK v1.8.0_91.
        -1 unit 0m 19s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91.
        -1 unit 106m 41s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91.
        -1 unit 0m 15s hadoop-common in the patch failed with JDK v1.7.0_95.
        -1 unit 0m 21s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95.
        -1 unit 0m 22s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        134m 44s



        Reason Tests
        JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter
          hadoop.mapred.TestMiniMRChildTask



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:cf2ee45
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804278/MAPREDUCE-6628.006.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux f5407abf5baf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4b55642
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvninstall-root.txt
        compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-compile-root-jdk1.7.0_95.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        mvneclipse https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvneclipse-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
        mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt
        javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/diff-checkstyle-root.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        mvneclipse https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvneclipse-hadoop-common-project_hadoop-common.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/whitespace-eol.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
        javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 8s Maven dependency ordering for branch -1 mvninstall 0m 8s root in trunk failed. +1 compile 7m 19s trunk passed with JDK v1.8.0_91 -1 compile 0m 9s root in trunk failed with JDK v1.7.0_95. +1 checkstyle 1m 56s trunk passed -1 mvnsite 0m 16s hadoop-common in trunk failed. -1 mvnsite 0m 14s hadoop-mapreduce-client-core in trunk failed. -1 mvnsite 0m 14s hadoop-mapreduce-client-jobclient in trunk failed. -1 mvneclipse 0m 8s hadoop-common in trunk failed. -1 findbugs 0m 8s hadoop-common in trunk failed. -1 findbugs 0m 11s hadoop-mapreduce-client-core in trunk failed. -1 findbugs 0m 11s hadoop-mapreduce-client-jobclient in trunk failed. -1 javadoc 0m 7s hadoop-common in trunk failed with JDK v1.8.0_91. -1 javadoc 0m 9s hadoop-common in trunk failed with JDK v1.7.0_95. 0 mvndep 0m 9s Maven dependency ordering for patch -1 mvninstall 0m 8s hadoop-common in the patch failed. -1 mvninstall 0m 12s hadoop-mapreduce-client-core in the patch failed. -1 mvninstall 0m 11s hadoop-mapreduce-client-jobclient in the patch failed. +1 compile 6m 48s the patch passed with JDK v1.8.0_91 +1 javac 6m 48s the patch passed -1 compile 0m 10s root in the patch failed with JDK v1.7.0_95. -1 javac 0m 10s root in the patch failed with JDK v1.7.0_95. -1 checkstyle 1m 26s root: patch generated 32 new + 232 unchanged - 4 fixed = 264 total (was 236) -1 mvnsite 0m 12s hadoop-common in the patch failed. -1 mvnsite 0m 14s hadoop-mapreduce-client-core in the patch failed. -1 mvnsite 0m 13s hadoop-mapreduce-client-jobclient in the patch failed. -1 mvneclipse 0m 8s hadoop-common in the patch failed. -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 8s hadoop-common in the patch failed. -1 findbugs 0m 12s hadoop-mapreduce-client-core in the patch failed. -1 findbugs 0m 10s hadoop-mapreduce-client-jobclient in the patch failed. +1 javadoc 1m 29s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91 with JDK v1.8.0_91 generated 0 new + 2497 unchanged - 12 fixed = 2497 total (was 2509) -1 javadoc 0m 8s hadoop-common in the patch failed with JDK v1.8.0_91. +1 javadoc 0m 22s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. +1 javadoc 0m 12s hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.8.0_91. -1 javadoc 0m 9s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 0m 7s hadoop-common in the patch failed with JDK v1.8.0_91. -1 unit 0m 19s hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91. -1 unit 106m 41s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. -1 unit 0m 15s hadoop-common in the patch failed with JDK v1.7.0_95. -1 unit 0m 21s hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. -1 unit 0m 22s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 134m 44s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.mapred.TestMRCJCFileOutputCommitter   hadoop.mapred.TestMiniMRChildTask Subsystem Report/Notes Docker Image:yetus/hadoop:cf2ee45 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804278/MAPREDUCE-6628.006.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f5407abf5baf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b55642 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvninstall-root.txt compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-compile-root-jdk1.7.0_95.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt mvneclipse https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvneclipse-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/diff-checkstyle-root.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt mvneclipse https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvneclipse-hadoop-common-project_hadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt javadoc https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 14s Maven dependency ordering for branch
        +1 mvninstall 6m 40s trunk passed
        +1 compile 6m 54s trunk passed
        +1 checkstyle 1m 29s trunk passed
        +1 mvnsite 1m 51s trunk passed
        +1 mvneclipse 0m 43s trunk passed
        +1 findbugs 2m 28s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 15s Maven dependency ordering for patch
        +1 mvninstall 1m 22s the patch passed
        +1 compile 6m 46s the patch passed
        +1 javac 6m 46s the patch passed
        -1 checkstyle 1m 29s root: The patch generated 32 new + 232 unchanged - 4 fixed = 264 total (was 236)
        +1 mvnsite 1m 51s the patch passed
        +1 mvneclipse 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 56s the patch passed
        +1 javadoc 0m 46s hadoop-common in the patch passed.
        +1 javadoc 0m 21s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508)
        +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 unit 7m 10s hadoop-common in the patch passed.
        +1 unit 2m 4s hadoop-mapreduce-client-core in the patch passed.
        +1 unit 117m 9s hadoop-mapreduce-client-jobclient in the patch passed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        166m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804278/MAPREDUCE-6628.006.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 86f931470f41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 3ca4d6d
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 40s trunk passed +1 compile 6m 54s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 43s trunk passed +1 findbugs 2m 28s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 1m 22s the patch passed +1 compile 6m 46s the patch passed +1 javac 6m 46s the patch passed -1 checkstyle 1m 29s root: The patch generated 32 new + 232 unchanged - 4 fixed = 264 total (was 236) +1 mvnsite 1m 51s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 56s the patch passed +1 javadoc 0m 46s hadoop-common in the patch passed. +1 javadoc 0m 21s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508) +1 javadoc 0m 13s hadoop-mapreduce-client-jobclient in the patch passed. +1 unit 7m 10s hadoop-common in the patch passed. +1 unit 2m 4s hadoop-mapreduce-client-core in the patch passed. +1 unit 117m 9s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 166m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804278/MAPREDUCE-6628.006.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 86f931470f41 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3ca4d6d Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6687/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 8m 9s trunk passed
        +1 compile 8m 3s trunk passed
        +1 checkstyle 1m 30s trunk passed
        +1 mvnsite 1m 51s trunk passed
        +1 mvneclipse 0m 45s trunk passed
        +1 findbugs 2m 35s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 7m 3s the patch passed
        +1 javac 7m 3s the patch passed
        +1 checkstyle 1m 27s root: The patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236)
        +1 mvnsite 1m 50s the patch passed
        +1 mvneclipse 0m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 5s the patch passed
        +1 javadoc 0m 53s hadoop-common in the patch passed.
        +1 javadoc 0m 23s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508)
        +1 javadoc 0m 14s hadoop-mapreduce-client-jobclient in the patch passed.
        -1 unit 7m 53s hadoop-common in the patch failed.
        +1 unit 2m 18s hadoop-mapreduce-client-core in the patch passed.
        -1 unit 117m 26s hadoop-mapreduce-client-jobclient in the patch failed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        171m 12s



        Reason Tests
        Failed junit tests hadoop.security.TestGroupsCaching
          hadoop.fs.slive.TestSlive



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825140/MAPREDUCE-6628.007.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 11f3482f5527 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / c37346d
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 8m 9s trunk passed +1 compile 8m 3s trunk passed +1 checkstyle 1m 30s trunk passed +1 mvnsite 1m 51s trunk passed +1 mvneclipse 0m 45s trunk passed +1 findbugs 2m 35s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 7m 3s the patch passed +1 javac 7m 3s the patch passed +1 checkstyle 1m 27s root: The patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236) +1 mvnsite 1m 50s the patch passed +1 mvneclipse 0m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 5s the patch passed +1 javadoc 0m 53s hadoop-common in the patch passed. +1 javadoc 0m 23s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508) +1 javadoc 0m 14s hadoop-mapreduce-client-jobclient in the patch passed. -1 unit 7m 53s hadoop-common in the patch failed. +1 unit 2m 18s hadoop-mapreduce-client-core in the patch passed. -1 unit 117m 26s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 171m 12s Reason Tests Failed junit tests hadoop.security.TestGroupsCaching   hadoop.fs.slive.TestSlive Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825140/MAPREDUCE-6628.007.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 11f3482f5527 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c37346d Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6691/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        The test failures seem to be unrelated to this patch. Can a committer please commit this Jira? Thanks.

        Show
        masokan Mariappan Asokan added a comment - The test failures seem to be unrelated to this patch. Can a committer please commit this Jira? Thanks.
        Hide
        chris.douglas Chris Douglas added a comment -

        Mariappan Asokan thank you for your patience with this.

        The unit test looks useful for debugging, but it doesn't actually verify the fix. As written, it's also expensive to run (starts a cluster) and relies on a platform-dependent scan of /proc/self/status, rather than using java.lang.management APIs. That said, unit testing this corner of MapReduce is not straightforward, and your posted results demonstrate both the issue and the fix. We can commit this without a MR test.

        Would it be possible to write a short unit test for CryptoOutputStream verifying the new closeOutputStream semantics? This should be very straightforward in Mockito, just checking that close behaves as expected when the flag is passed.

        It's unfortunate that we're switching behavior based on object reference equality, to check whether the stream was wrapped. As designed, I don't see a cleaner way to improve this without refactoring the crypto implementation.

        Show
        chris.douglas Chris Douglas added a comment - Mariappan Asokan thank you for your patience with this. The unit test looks useful for debugging, but it doesn't actually verify the fix. As written, it's also expensive to run (starts a cluster) and relies on a platform-dependent scan of /proc/self/status , rather than using java.lang.management APIs. That said, unit testing this corner of MapReduce is not straightforward, and your posted results demonstrate both the issue and the fix. We can commit this without a MR test. Would it be possible to write a short unit test for CryptoOutputStream verifying the new closeOutputStream semantics? This should be very straightforward in Mockito, just checking that close behaves as expected when the flag is passed. It's unfortunate that we're switching behavior based on object reference equality, to check whether the stream was wrapped. As designed, I don't see a cleaner way to improve this without refactoring the crypto implementation.
        Hide
        masokan Mariappan Asokan added a comment -

        Chris, thanks for looking at the patch. I agree with you. I will create a new unit test for CryptoOutputStream and upload a new patch.

        Show
        masokan Mariappan Asokan added a comment - Chris, thanks for looking at the patch. I agree with you. I will create a new unit test for CryptoOutputStream and upload a new patch.
        Hide
        masokan Mariappan Asokan added a comment -

        Uploaded a new patch with a unit test that tests closing/not closing the underlying OutputStream of CryptoOutputStream.

        Show
        masokan Mariappan Asokan added a comment - Uploaded a new patch with a unit test that tests closing/not closing the underlying OutputStream of CryptoOutputStream .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 23s Maven dependency ordering for branch
        +1 mvninstall 7m 14s trunk passed
        +1 compile 7m 1s trunk passed
        +1 checkstyle 1m 29s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 2m 12s trunk passed
        +1 javadoc 1m 11s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 13s the patch passed
        +1 compile 8m 1s the patch passed
        +1 javac 8m 1s the patch passed
        -1 checkstyle 1m 35s root: The patch generated 1 new + 233 unchanged - 4 fixed = 234 total (was 237)
        +1 mvnsite 1m 31s the patch passed
        +1 mvneclipse 0m 27s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 52s the patch passed
        +1 javadoc 0m 47s hadoop-common in the patch passed.
        +1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508)
        +1 unit 8m 58s hadoop-common in the patch passed.
        +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        52m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827773/MAPREDUCE-6628.008.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux ab66468d4418 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d4d0768
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/artifact/patchprocess/diff-checkstyle-root.txt
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 23s Maven dependency ordering for branch +1 mvninstall 7m 14s trunk passed +1 compile 7m 1s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 12s trunk passed +1 javadoc 1m 11s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 8m 1s the patch passed +1 javac 8m 1s the patch passed -1 checkstyle 1m 35s root: The patch generated 1 new + 233 unchanged - 4 fixed = 234 total (was 237) +1 mvnsite 1m 31s the patch passed +1 mvneclipse 0m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 52s the patch passed +1 javadoc 0m 47s hadoop-common in the patch passed. +1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508) +1 unit 8m 58s hadoop-common in the patch passed. +1 unit 2m 13s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 52m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827773/MAPREDUCE-6628.008.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ab66468d4418 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4d0768 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6711/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        Got rid of checkstyle warning.

        Show
        masokan Mariappan Asokan added a comment - Got rid of checkstyle warning.
        Hide
        chris.douglas Chris Douglas added a comment -

        +1

        I committed this. Thanks Mariappan Asokan

        Show
        chris.douglas Chris Douglas added a comment - +1 I committed this. Thanks Mariappan Asokan
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10420 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10420/)
        MAPREDUCE-6628. Potential memory leak in CryptoOutputStream. Contributed (cdouglas: rev 9f192cc5ac4a6145e2eeaecba0a754d31e601898)

        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto/CryptoFSDataOutputStream.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
        • (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoOutputStreamClosing.java
        • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10420 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10420/ ) MAPREDUCE-6628 . Potential memory leak in CryptoOutputStream. Contributed (cdouglas: rev 9f192cc5ac4a6145e2eeaecba0a754d31e601898) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/crypto/CryptoFSDataOutputStream.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/CryptoUtils.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java (add) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoOutputStreamClosing.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoOutputStream.java
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 15s Maven dependency ordering for branch
        +1 mvninstall 8m 13s trunk passed
        +1 compile 8m 35s trunk passed
        +1 checkstyle 1m 34s trunk passed
        +1 mvnsite 1m 33s trunk passed
        +1 mvneclipse 0m 28s trunk passed
        +1 findbugs 2m 32s trunk passed
        +1 javadoc 1m 15s trunk passed
        0 mvndep 0m 16s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 7m 55s the patch passed
        +1 javac 7m 55s the patch passed
        +1 checkstyle 1m 33s root: The patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236)
        +1 mvnsite 1m 30s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 2m 47s the patch passed
        +1 javadoc 0m 49s hadoop-common in the patch passed.
        +1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508)
        +1 unit 8m 27s hadoop-common in the patch passed.
        +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        53m 42s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827797/MAPREDUCE-6628.009.patch
        JIRA Issue MAPREDUCE-6628
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3268efb2836c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4b6d795
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6714/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: .
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6714/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 15s Maven dependency ordering for branch +1 mvninstall 8m 13s trunk passed +1 compile 8m 35s trunk passed +1 checkstyle 1m 34s trunk passed +1 mvnsite 1m 33s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 2m 32s trunk passed +1 javadoc 1m 15s trunk passed 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 7m 55s the patch passed +1 javac 7m 55s the patch passed +1 checkstyle 1m 33s root: The patch generated 0 new + 232 unchanged - 4 fixed = 232 total (was 236) +1 mvnsite 1m 30s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 47s the patch passed +1 javadoc 0m 49s hadoop-common in the patch passed. +1 javadoc 0m 22s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core generated 0 new + 2496 unchanged - 12 fixed = 2496 total (was 2508) +1 unit 8m 27s hadoop-common in the patch passed. +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 53m 42s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827797/MAPREDUCE-6628.009.patch JIRA Issue MAPREDUCE-6628 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3268efb2836c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4b6d795 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6714/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: . Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6714/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        masokan Mariappan Asokan added a comment -

        Thanks Chris.

        Show
        masokan Mariappan Asokan added a comment - Thanks Chris.

          People

          • Assignee:
            masokan Mariappan Asokan
            Reporter:
            masokan Mariappan Asokan
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development