Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Possible resource leakage in HDFS.

      1. HDFS-1990.patch
        0.6 kB
        ramkrishna.s.vasudevan
      2. HDFS-1990.patch
        0.7 kB
        Uma Maheswara Rao G

        Activity

        Hide
        Aaron T. Myers added a comment -

        Hi Ramkrishna, can you provide any more specific details about exactly what resources appear to be leaking under what circumstances? Thanks a lot.

        Show
        Aaron T. Myers added a comment - Hi Ramkrishna, can you provide any more specific details about exactly what resources appear to be leaking under what circumstances? Thanks a lot.
        Hide
        ramkrishna.s.vasudevan added a comment -

        In BlockReceiver.java

        In close() api

        try
                {
        	  checksumOut.flush();
                  if (datanode.syncOnClose && (cout instanceof FileOutputStream)) {
        	    ((FileOutputStream) cout).getChannel().force(true);
        	  }
        	 checksumOut.close();
               } catch (IOException e) {
        	  ioe = e;
        	}
        

        we are trying to close the streams in try block.
        If some exception occurs we will not be closing the streams which may lead to resource leak.
        Similar thing can happen in

        try{
          if (out != null) {
           out.flush();
          
          if (datanode.syncOnClose && (out instanceof FileOutputStream)) {
              ((FileOutputStream) out).getChannel().force(true);
        	}
        	out.close();
          } catch (IOException e) {
             ioe = e;
             }
        
        Show
        ramkrishna.s.vasudevan added a comment - In BlockReceiver.java In close() api try { checksumOut.flush(); if (datanode.syncOnClose && (cout instanceof FileOutputStream)) { ((FileOutputStream) cout).getChannel().force(true); } checksumOut.close(); } catch (IOException e) { ioe = e; } we are trying to close the streams in try block. If some exception occurs we will not be closing the streams which may lead to resource leak. Similar thing can happen in try{ if (out != null) { out.flush(); if (datanode.syncOnClose && (out instanceof FileOutputStream)) { ((FileOutputStream) out).getChannel().force(true); } out.close(); } catch (IOException e) { ioe = e; }
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480948/HDFS-1990.patch
        against trunk revision 1128987.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480948/HDFS-1990.patch against trunk revision 1128987. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        Handled the stream closures as per the discussion in below issue.
        https://issues.apache.org/jira/browse/MAPREDUCE-2243.
        also in https://issues.apache.org/jira/browse/HDFS-1753

        Regenerated the patch against to latest trunk!

        Show
        Uma Maheswara Rao G added a comment - Handled the stream closures as per the discussion in below issue. https://issues.apache.org/jira/browse/MAPREDUCE-2243 . also in https://issues.apache.org/jira/browse/HDFS-1753 Regenerated the patch against to latest trunk!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12485287/HDFS-1990.patch
        against trunk revision 1141767.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/877//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/877//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/877//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485287/HDFS-1990.patch against trunk revision 1141767. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/877//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/877//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/877//console This message is automatically generated.
        Hide
        Uma Maheswara Rao G added a comment -

        difficult to assert the stream closes alone.
        Since all existing tests are passing with this changes , no need to have special tests for this changes.

        Show
        Uma Maheswara Rao G added a comment - difficult to assert the stream closes alone. Since all existing tests are passing with this changes , no need to have special tests for this changes.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Uma!

        Show
        Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/774/)
        HDFS-1990. Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/774/ ) HDFS-1990 . Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        Eli Collins added a comment -

        Nite: the finally clause should not be on it's own line. Otherwise looks great.

        Show
        Eli Collins added a comment - Nite: the finally clause should not be on it's own line. Otherwise looks great.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #717 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/717/)
        HDFS-1990. Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #717 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/717/ ) HDFS-1990 . Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java

          People

          • Assignee:
            Uma Maheswara Rao G
            Reporter:
            ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development