Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Possible resource leakage in HDFS.

      1. HDFS-1990.patch
        0.7 kB
        Uma Maheswara Rao G
      2. HDFS-1990.patch
        0.6 kB
        ramkrishna.s.vasudevan

        Activity

        Hide
        atm Aaron T. Myers added a comment -

        Hi Ramkrishna, can you provide any more specific details about exactly what resources appear to be leaking under what circumstances? Thanks a lot.

        Show
        atm Aaron T. Myers added a comment - Hi Ramkrishna, can you provide any more specific details about exactly what resources appear to be leaking under what circumstances? Thanks a lot.
        Hide
        ram_krish ramkrishna.s.vasudevan added a comment -

        In BlockReceiver.java

        In close() api

        try
                {
        	  checksumOut.flush();
                  if (datanode.syncOnClose && (cout instanceof FileOutputStream)) {
        	    ((FileOutputStream) cout).getChannel().force(true);
        	  }
        	 checksumOut.close();
               } catch (IOException e) {
        	  ioe = e;
        	}
        

        we are trying to close the streams in try block.
        If some exception occurs we will not be closing the streams which may lead to resource leak.
        Similar thing can happen in

        try{
          if (out != null) {
           out.flush();
          
          if (datanode.syncOnClose && (out instanceof FileOutputStream)) {
              ((FileOutputStream) out).getChannel().force(true);
        	}
        	out.close();
          } catch (IOException e) {
             ioe = e;
             }
        
        Show
        ram_krish ramkrishna.s.vasudevan added a comment - In BlockReceiver.java In close() api try { checksumOut.flush(); if (datanode.syncOnClose && (cout instanceof FileOutputStream)) { ((FileOutputStream) cout).getChannel().force(true); } checksumOut.close(); } catch (IOException e) { ioe = e; } we are trying to close the streams in try block. If some exception occurs we will not be closing the streams which may lead to resource leak. Similar thing can happen in try{ if (out != null) { out.flush(); if (datanode.syncOnClose && (out instanceof FileOutputStream)) { ((FileOutputStream) out).getChannel().force(true); } out.close(); } catch (IOException e) { ioe = e; }
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480948/HDFS-1990.patch
        against trunk revision 1128987.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480948/HDFS-1990.patch against trunk revision 1128987. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/665//console This message is automatically generated.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        Handled the stream closures as per the discussion in below issue.
        https://issues.apache.org/jira/browse/MAPREDUCE-2243.
        also in https://issues.apache.org/jira/browse/HDFS-1753

        Regenerated the patch against to latest trunk!

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - Handled the stream closures as per the discussion in below issue. https://issues.apache.org/jira/browse/MAPREDUCE-2243 . also in https://issues.apache.org/jira/browse/HDFS-1753 Regenerated the patch against to latest trunk!
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12485287/HDFS-1990.patch
        against trunk revision 1141767.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/877//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/877//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/877//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485287/HDFS-1990.patch against trunk revision 1141767. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/877//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/877//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/877//console This message is automatically generated.
        Hide
        umamaheswararao Uma Maheswara Rao G added a comment -

        difficult to assert the stream closes alone.
        Since all existing tests are passing with this changes , no need to have special tests for this changes.

        Show
        umamaheswararao Uma Maheswara Rao G added a comment - difficult to assert the stream closes alone. Since all existing tests are passing with this changes , no need to have special tests for this changes.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        +1 patch looks good.

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
        Hide
        szetszwo Tsz Wo Nicholas Sze added a comment -

        I have committed this. Thanks, Uma!

        Show
        szetszwo Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Uma!
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/774/)
        HDFS-1990. Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #774 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/774/ ) HDFS-1990 . Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Hide
        eli Eli Collins added a comment -

        Nite: the finally clause should not be on it's own line. Otherwise looks great.

        Show
        eli Eli Collins added a comment - Nite: the finally clause should not be on it's own line. Otherwise looks great.
        Hide
        hudson Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #717 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/717/)
        HDFS-1990. Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
        Show
        hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #717 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/717/ ) HDFS-1990 . Fix resource leaks in BlockReceiver.close(). Contributed by Uma Maheswara Rao G szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1143147 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java

          People

          • Assignee:
            umamaheswararao Uma Maheswara Rao G
            Reporter:
            ram_krish ramkrishna.s.vasudevan
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development