Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1118

DFSOutputStream socket leak when cannot connect to DataNode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.1, 0.20.2, 0.20-append, 0.21.0
    • Fix Version/s: 0.20-append, 0.20.205.0, 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The offending code is in DFSOutputStream.nextBlockOutputStream

      This function retries several times to call createBlockOutputStream. Each time when it fails, it leaves a Socket object in DFSOutputStream.s.
      That object is never closed, but overwritten the next time createBlockOutputStream is called.

      1. HDFS-1118.1.patch
        0.5 kB
        Zheng Shao
      2. HDFS-1118.2.patch
        1 kB
        Zheng Shao
      3. trunkPatch.txt
        1 kB
        dhruba borthakur
      4. hdfs-1118.20s.patch
        3 kB
        Suresh Srinivas

        Activity

        Zheng Shao created issue -
        Hide
        Todd Lipcon added a comment -

        Great catch, Zheng! I've seen leaked sockets before but never been able to figure out what the cause was.

        Show
        Todd Lipcon added a comment - Great catch, Zheng! I've seen leaked sockets before but never been able to figure out what the cause was.
        Hide
        Zheng Shao added a comment -

        This patch does not have a unit test yet. Not sure what is the best way to write one for this.

        Show
        Zheng Shao added a comment - This patch does not have a unit test yet. Not sure what is the best way to write one for this.
        Zheng Shao made changes -
        Field Original Value New Value
        Attachment HDFS-1118.1.patch [ 12443148 ]
        Zheng Shao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12443148/HDFS-1118.1.patch
        against trunk revision 939091.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/165/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443148/HDFS-1118.1.patch against trunk revision 939091. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/165/console This message is automatically generated.
        Hide
        dhruba borthakur added a comment -

        +1 good catch. awesome!

        Show
        dhruba borthakur added a comment - +1 good catch. awesome!
        Hide
        Zheng Shao added a comment -

        Moved the cleanup to finally section.

        Show
        Zheng Shao added a comment - Moved the cleanup to finally section.
        Zheng Shao made changes -
        Attachment HDFS-1118.2.patch [ 12443248 ]
        Zheng Shao made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Zheng Shao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Zheng Shao made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Zheng Shao made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Todd Lipcon added a comment -

        Hey Zheng. I was just running into this problem and then remembered that you put this patch up a while back.

        Why not just add an IOUtils.closeSocket() call inside the IOException catch clause, instead of adding the finally {} clause?

        Show
        Todd Lipcon added a comment - Hey Zheng. I was just running into this problem and then remembered that you put this patch up a while back. Why not just add an IOUtils.closeSocket() call inside the IOException catch clause, instead of adding the finally {} clause?
        Hide
        Todd Lipcon added a comment -

        Marking this for 0.20-append

        Show
        Todd Lipcon added a comment - Marking this for 0.20-append
        Todd Lipcon made changes -
        Fix Version/s 0.20-append [ 12315103 ]
        Affects Version/s 0.20-append [ 12315103 ]
        Hide
        dhruba borthakur added a comment -

        Code looks good to me. I will commit this to trunk.

        Show
        dhruba borthakur added a comment - Code looks good to me. I will commit this to trunk.
        Hide
        dhruba borthakur added a comment -

        Patch for apache hadop trunk.

        Todd: the finally clause should not hurt, will catch all types of exceptions and cleanup the socket.

        Looks ok without a unit test, because the code is simple and I agree with Zheng that it is difficult to write a unit test

        Show
        dhruba borthakur added a comment - Patch for apache hadop trunk. Todd: the finally clause should not hurt, will catch all types of exceptions and cleanup the socket. Looks ok without a unit test, because the code is simple and I agree with Zheng that it is difficult to write a unit test
        dhruba borthakur made changes -
        Attachment trunkPatch.txt [ 12447550 ]
        Hide
        dhruba borthakur added a comment -

        I just committed this. Thanks Zheng.

        Show
        dhruba borthakur added a comment - I just committed this. Thanks Zheng.
        dhruba borthakur made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314241 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #315 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/315/)
        HDFS-1118. Fix socketleak on DFSClient.
        (Zheng Shao via dhruba)

        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #315 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/315/ ) HDFS-1118 . Fix socketleak on DFSClient. (Zheng Shao via dhruba)
        Hide
        Aaron Baff added a comment -

        This bug also affects version 0.21.0

        Show
        Aaron Baff added a comment - This bug also affects version 0.21.0
        Aaron Baff made changes -
        Affects Version/s 0.21.0 [ 12314046 ]
        Suresh Srinivas made changes -
        Fix Version/s 0.20.205.0 [ 12316392 ]
        Hide
        Suresh Srinivas added a comment -

        Merge patch from 0.20-append to 0.20-security

        Show
        Suresh Srinivas added a comment - Merge patch from 0.20-append to 0.20-security
        Suresh Srinivas made changes -
        Attachment hdfs-1118.20s.patch [ 12491266 ]
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for hdfs-1118.20s.patch

        Show
        Jitendra Nath Pandey added a comment - +1 for hdfs-1118.20s.patch
        Hide
        Suresh Srinivas added a comment -

        I have committed the patch to 0.20-security branch.

        Show
        Suresh Srinivas added a comment - I have committed the patch to 0.20-security branch.
        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        1d 19h 30m 2 Zheng Shao 30/Apr/10 21:08
        Open Open Patch Available Patch Available
        19m 41s 3 Zheng Shao 30/Apr/10 21:08
        Patch Available Patch Available Resolved Resolved
        50d 12h 24m 1 dhruba borthakur 20/Jun/10 09:32
        Resolved Resolved Closed Closed
        485d 15h 53m 1 Matt Foley 19/Oct/11 01:26

          People

          • Assignee:
            Zheng Shao
            Reporter:
            Zheng Shao
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development