Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7256

Resource leak during failure scenario of closing of resources.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Problem Statement:
      ===============
      There are chances of resource leak and stream not getting closed
      Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
      Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.

      Scenario
      =======
      During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
      Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.

      Solution:
      =======
      While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.

      Additionally there are places where we try to close a resource in the catch block.
      If this close fails, we just throw and come out of the current flow.
      In order to avoid this, we can carry out the close operation in the finally block.

      Probable reasons for getting RunTimeExceptions:
      =====================================
      We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.

      1. HADOOP-7256-patch-1.patch
        3 kB
        ramkrishna.s.vasudevan
      2. HADOOP-7256-patch-2.patch
        3 kB
        ramkrishna.s.vasudevan
      3. HADOOP-7256.patch
        4 kB
        Harsh J

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479607/HADOOP-7256-patch-1.patch
          against trunk revision 1104426.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479607/HADOOP-7256-patch-1.patch against trunk revision 1104426. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/468//console This message is automatically generated.
          Hide
          tlipcon Todd Lipcon added a comment -
          • looks like there are some hard tabs, please configure your editor to use spaces per our coding guidelines
          • in the newly added LOG.debug line, please include some text. eg LOG.debug("Ignoring error closing socket", ignored);
          • in the test case, please add a fail("Didn't throw") line after the IOUtils.copyBytes call. That will make sure that the exception is properly propagated and not swallowed.
          • Probably better to separate the test in two – one for the case where the inputstream throws, and another for the case when the output stream throws.
          Show
          tlipcon Todd Lipcon added a comment - looks like there are some hard tabs, please configure your editor to use spaces per our coding guidelines in the newly added LOG.debug line, please include some text. eg LOG.debug("Ignoring error closing socket", ignored); in the test case, please add a fail("Didn't throw") line after the IOUtils.copyBytes call. That will make sure that the exception is properly propagated and not swallowed. Probably better to separate the test in two – one for the case where the inputstream throws, and another for the case when the output stream throws.
          Hide
          tlipcon Todd Lipcon added a comment -
          • How is this an incompatible change?
          • Please don't click the "reviewed" checkbox until a committer has reviewed.
          Show
          tlipcon Todd Lipcon added a comment - How is this an incompatible change? Please don't click the "reviewed" checkbox until a committer has reviewed.
          Hide
          ram_krish ramkrishna.s.vasudevan added a comment -

          Hi Todd,

          Thanks for the comments. When I had submitted this issue my IDE had the wrong formatter. Now I have changed them.

          I will fix all your comments and resubmit it and also will take care this formatting and check box problem doesnt come here then.

          Show
          ram_krish ramkrishna.s.vasudevan added a comment - Hi Todd, Thanks for the comments. When I had submitted this issue my IDE had the wrong formatter. Now I have changed them. I will fix all your comments and resubmit it and also will take care this formatting and check box problem doesnt come here then.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12480943/HADOOP-7256-patch-2.patch
          against trunk revision 1128789.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480943/HADOOP-7256-patch-2.patch against trunk revision 1128789. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/543//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Rebased patch to trunk. Changes lgtm, +1. Awaiting jenkins result.

          Show
          qwertymaniac Harsh J added a comment - Rebased patch to trunk. Changes lgtm, +1. Awaiting jenkins result.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12546173/HADOOP-7256.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1500//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1500//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12546173/HADOOP-7256.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1500//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1500//console This message is automatically generated.
          Hide
          qwertymaniac Harsh J added a comment -

          Forgot to mention, I'd cleaned up some extra whitespace issues around the test and closed a RAFile the test was using without closing (Eclipse warned a resource leak).

          Committing shortly to trunk.

          Show
          qwertymaniac Harsh J added a comment - Forgot to mention, I'd cleaned up some extra whitespace issues around the test and closed a RAFile the test was using without closing (Eclipse warned a resource leak). Committing shortly to trunk.
          Hide
          qwertymaniac Harsh J added a comment -

          Committed to trunk. Thanks Ram!

          Show
          qwertymaniac Harsh J added a comment - Committed to trunk. Thanks Ram!
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2756 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2756/)
          HADOOP-7256. Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2756 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2756/ ) HADOOP-7256 . Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2819 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2819/)
          HADOOP-7256. Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2819 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2819/ ) HADOOP-7256 . Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2778 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2778/)
          HADOOP-7256. Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893)

          Result = FAILURE
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2778 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2778/ ) HADOOP-7256 . Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893) Result = FAILURE harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/)
          HADOOP-7256. Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1174 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1174/ ) HADOOP-7256 . Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/)
          HADOOP-7256. Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893)

          Result = SUCCESS
          harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Show
          hudson Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1205 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1205/ ) HADOOP-7256 . Resource leak during failure scenario of closing of resources. Contributed by Ramkrishna S. Vasudevan. (harsh) (Revision 1388893) Result = SUCCESS harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388893 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestIOUtils.java
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Wei-Chiu Chuang and Harsh J, can we backport this to branch-2? I think this fix is valuable and compatible because IOUtils.java is @Evolving.

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Wei-Chiu Chuang and Harsh J , can we backport this to branch-2? I think this fix is valuable and compatible because IOUtils.java is @Evolving .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'll backport this to branch-2 tomorrow if there are not any objections.

          Show
          ajisakaa Akira Ajisaka added a comment - I'll backport this to branch-2 tomorrow if there are not any objections.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Backported to branch-2.

          Show
          ajisakaa Akira Ajisaka added a comment - Backported to branch-2.

            People

            • Assignee:
              ram_krish ramkrishna.s.vasudevan
              Reporter:
              ram_krish ramkrishna.s.vasudevan
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 8h
                8h
                Remaining:
                Remaining Estimate - 8h
                8h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development