Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-7256

Resource leak during failure scenario of closing of resources.

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.20.2
    • 2.9.0, 3.0.0-alpha1
    • util
    • None
    • Reviewed

    Description

      Problem Statement:
      ===============
      There are chances of resource leak and stream not getting closed
      Take the case when after copying data we try to close the Input and output stream followed by closing of the socket.
      Suppose an exception occurs while closing the input stream(due to runtime exception) then the subsequent operations of closing the output stream and socket may not happen and there is a chance of resource leak.

      Scenario
      =======
      During long run of map reduce jobs, the copyFromLocalFile() api is getting called.
      Here we found some exceptions happening. As a result of this we found the lsof value raising leading to resource leak.

      Solution:
      =======
      While doing a close operation of any resource catch the RuntimeException also rather than catching the IOException alone.

      Additionally there are places where we try to close a resource in the catch block.
      If this close fails, we just throw and come out of the current flow.
      In order to avoid this, we can carry out the close operation in the finally block.

      Probable reasons for getting RunTimeExceptions:
      =====================================
      We may get runtime exception from customised hadoop streams like FSDataOutputStream.close() . So better to handle RunTimeExceptions also.

      Attachments

        1. HADOOP-7256.patch
          4 kB
          Harsh J
        2. HADOOP-7256-patch-2.patch
          3 kB
          ramkrishna.s.vasudevan
        3. HADOOP-7256-patch-1.patch
          3 kB
          ramkrishna.s.vasudevan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ram_krish ramkrishna.s.vasudevan
            ram_krish ramkrishna.s.vasudevan
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 8h
                8h
                Remaining:
                Remaining Estimate - 8h
                8h
                Logged:
                Time Spent - Not Specified
                Not Specified

                Issue deployment