Hadoop Common
  1. Hadoop Common
  2. HADOOP-7090

Possible resource leaks in hadoop core code

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: fs/s3, io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      resource leak

      Description

      It is always a good practice to close the IO streams in a finally block..

      For example, look at the following piece of code in the Writer class of BloomMapFile

      BloomMapFile .java
          public synchronized void close() throws IOException {
            super.close();
            DataOutputStream out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
            bloomFilter.write(out);
            out.flush();
            out.close();
          }
      

      If an exception occurs during fs.create or on any other line, out.close() will not be executed..

      The following can reduce the scope of resorce leaks..

      BloomMapFile .java
          public synchronized void close() throws IOException {
            super.close();
            DataOutputStream out = null;
            try{
                out = fs.create(new Path(dir, BLOOM_FILE_NAME), true);
                bloomFilter.write(out);
                out.flush();
            }finally{
      	 IOUtils.closeStream(out);
          }
      
      1. HADOOP-7090.0.patch
        5 kB
        Uma Maheswara Rao G
      2. HADOOP-7090.patch
        5 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #739 (See https://builds.apache.org/job/Hadoop-Common-trunk/739/)
        HADOOP-7090. Fix resource leaks in s3.INode, BloomMapFile, WritableUtils and CBZip2OutputStream. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143149
        Files :

        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/WritableUtils.java
        • /hadoop/common/trunk/common/CHANGES.txt
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/s3/INode.java
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/BloomMapFile.java
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #739 (See https://builds.apache.org/job/Hadoop-Common-trunk/739/ ) HADOOP-7090 . Fix resource leaks in s3.INode, BloomMapFile, WritableUtils and CBZip2OutputStream. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143149 Files : /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/WritableUtils.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/s3/INode.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/BloomMapFile.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #672 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/672/)
        HADOOP-7090. Fix resource leaks in s3.INode, BloomMapFile, WritableUtils and CBZip2OutputStream. Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143149
        Files :

        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/WritableUtils.java
        • /hadoop/common/trunk/common/CHANGES.txt
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/s3/INode.java
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/BloomMapFile.java
        • /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #672 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/672/ ) HADOOP-7090 . Fix resource leaks in s3.INode, BloomMapFile, WritableUtils and CBZip2OutputStream. Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143149 Files : /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/WritableUtils.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/java/org/apache/hadoop/fs/s3/INode.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/BloomMapFile.java /hadoop/common/trunk/common/src/java/org/apache/hadoop/io/compress/bzip2/CBZip2OutputStream.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Uma!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma!
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Uma Maheswara Rao G added a comment -

        difficult to assert the stream closes alone.
        Since all existing tests are passing with this changes , no need to have special tests for this changes.

        Show
        Uma Maheswara Rao G added a comment - difficult to assert the stream closes alone. Since all existing tests are passing with this changes , no need to have special tests for this changes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12485286/HADOOP-7090.patch
        against trunk revision 1142556.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485286/HADOOP-7090.patch against trunk revision 1142556. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/693//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Handled the stream closures as per the discussion in below issue.
        https://issues.apache.org/jira/browse/MAPREDUCE-2243.
        also in https://issues.apache.org/jira/browse/HDFS-1753

        Show
        Uma Maheswara Rao G added a comment - Handled the stream closures as per the discussion in below issue. https://issues.apache.org/jira/browse/MAPREDUCE-2243 . also in https://issues.apache.org/jira/browse/HDFS-1753
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480947/HADOOP-7090.0.patch
        against trunk revision 1128789.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480947/HADOOP-7090.0.patch against trunk revision 1128789. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/544//console This message is automatically generated.
        Hide
        Owen O'Malley added a comment -

        See my comment at https://issues.apache.org/jira/browse/MAPREDUCE-2243?focusedCommentId=12977849&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12977849

        The nominal close must be done in the try block. Only the exceptional close must be done in a catch block.

        Show
        Owen O'Malley added a comment - See my comment at https://issues.apache.org/jira/browse/MAPREDUCE-2243?focusedCommentId=12977849&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12977849 The nominal close must be done in the try block. Only the exceptional close must be done in a catch block.
        Hide
        Gokul added a comment -


        I am collecting the similar cases in the code..

        Show
        Gokul added a comment - I am collecting the similar cases in the code..

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            Gokul
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development