Hadoop Common
  1. Hadoop Common
  2. HADOOP-6754

DefaultCodec.createOutputStream() leaks memory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DefaultCodec.createOutputStream() creates a new Compressor instance in each OutputStream. Even if the OutputStream is closed, this leaks memory.

      1. HADOOP-6754.patch
        1 kB
        Aaron Kimball
      2. HADOOP-6754.patch
        2 kB
        Tom White
      3. HADOOP-6754.2.patch
        1 kB
        Aaron Kimball
      4. CompressionBug.java
        1 kB
        Aaron Kimball

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #517 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/)

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #517 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/517/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #619 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/619/)
        HADOOP-6754. DefaultCodec.createOutputStream() leaks memory. Contributed by Aaron Kimball.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #619 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/619/ ) HADOOP-6754 . DefaultCodec.createOutputStream() leaks memory. Contributed by Aaron Kimball.
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Aaron!

        Show
        Tom White added a comment - I've just committed this. Thanks Aaron!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12459074/HADOOP-6754.patch
        against trunk revision 1071364.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459074/HADOOP-6754.patch against trunk revision 1071364. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/273//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12459074/HADOOP-6754.patch
        against trunk revision 1031948.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459074/HADOOP-6754.patch against trunk revision 1031948. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/79//console This message is automatically generated.
        Hide
        Tom White added a comment -

        +1 This would be good to add. I've regenerated the patch.

        Show
        Tom White added a comment - +1 This would be good to add. I've regenerated the patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445381/HADOOP-6754.2.patch
        against trunk revision 1031422.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/37//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445381/HADOOP-6754.2.patch against trunk revision 1031422. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/37//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445381/HADOOP-6754.2.patch
        against trunk revision 947747.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445381/HADOOP-6754.2.patch against trunk revision 947747. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/541/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        New patch with --no-prefix

        Show
        Aaron Kimball added a comment - New patch with --no-prefix
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12445379/HADOOP-6754.patch
        against trunk revision 947747.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/540/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12445379/HADOOP-6754.patch against trunk revision 947747. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/540/console This message is automatically generated.
        Hide
        Aaron Kimball added a comment -

        If this method should not be called, and there is a preferred method, then this one should at least log a warning informing the user that he's calling a bad API. Attached is a patch which explains the situation.

        No new tests because this does not modify the underlying behavior of the system.

        Show
        Aaron Kimball added a comment - If this method should not be called, and there is a preferred method, then this one should at least log a warning informing the user that he's calling a bad API. Attached is a patch which explains the situation. No new tests because this does not modify the underlying behavior of the system.
        Hide
        Hong Tang added a comment -

        DefaultCodec.createOutputStream(OutputStream out) would be useful when you do not use native compression library. Otherwise, a direct buffer would be created and leaked. The problem with direct buffer is that it is not accounted under garbage collection and thus you may exhaust the memory yet does not trigger gc.

        Show
        Hong Tang added a comment - DefaultCodec.createOutputStream(OutputStream out) would be useful when you do not use native compression library. Otherwise, a direct buffer would be created and leaked. The problem with direct buffer is that it is not accounted under garbage collection and thus you may exhaust the memory yet does not trigger gc.
        Hide
        Aaron Kimball added a comment -

        How is it not a bug that this leaks memory? If there is a preferred API which works correctly, then this one should be marked as deprecated and eventually removed.

        Show
        Aaron Kimball added a comment - How is it not a bug that this leaks memory? If there is a preferred API which works correctly, then this one should be marked as deprecated and eventually removed.
        Hide
        Hong Tang added a comment -

        Aaron, this is not a bug, you should use DefaultCodec.createOutputStream(OutputStream out, Compressor compressor).

        Show
        Hong Tang added a comment - Aaron, this is not a bug, you should use DefaultCodec.createOutputStream(OutputStream out, Compressor compressor).
        Hide
        Aaron Kimball added a comment -

        This short program illustrates the problem. The failing() method will terminate with an OutOfMemoryException. This occurs quickly with native libs enabled and less quickly (more iterations of the loop) with native libs disabled - but both of these leak.

        Show
        Aaron Kimball added a comment - This short program illustrates the problem. The failing() method will terminate with an OutOfMemoryException. This occurs quickly with native libs enabled and less quickly (more iterations of the loop) with native libs disabled - but both of these leak.

          People

          • Assignee:
            Aaron Kimball
            Reporter:
            Aaron Kimball
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development