Hadoop Common
  1. Hadoop Common
  2. HADOOP-6722

NetUtils.connect should check that it hasn't connected a socket to itself

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.205.0, 0.21.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      I had no idea this was possible, but it turns out that a TCP connection will be established in the rare case that the local side of the socket binds to the ephemeral port that you later try to connect to. This can present itself in very very rare occasion when an RPC client is trying to connect to a daemon running on the same node, but that daemon is down. To see what I'm talking about, run "while true ; do telnet localhost 60020 ; done" on a multicore box and wait several minutes.

      This can be easily detected in NetUtils.connect by making sure the local address/port is not equal to the remote address/port.

      1. hadoop-6722.txt
        4 kB
        Todd Lipcon
      2. HADOOP-6722.20s.patch
        4 kB
        Suresh Srinivas

        Activity

        Hide
        Todd Lipcon added a comment -

        Here's a patch which detects this situation and throws a ConnectException.

        The test case manufactures the problem by binding to an ephemeral port and then trying to connect to itself.

        (fwiw, I actually did run into this issue while testing hbase under failure injection scenarios - a client tried to open RPC to the hbase server, but got itself instead, and was very unhappy)

        Show
        Todd Lipcon added a comment - Here's a patch which detects this situation and throws a ConnectException. The test case manufactures the problem by binding to an ephemeral port and then trying to connect to itself. (fwiw, I actually did run into this issue while testing hbase under failure injection scenarios - a client tried to open RPC to the hbase server, but got itself instead, and was very unhappy)
        Hide
        Hong Tang added a comment -

        I was bit by this "feature" of TCP in my past life too.

        Show
        Hong Tang added a comment - I was bit by this "feature" of TCP in my past life too.
        Hide
        Allen Wittenauer added a comment -

        FWIW, I can't duplicate this on Solaris or Mac OS X, both of which are based upon BSD sockets. So I'm guessing this is a bug in the Linux TCP/IP stack.

        Show
        Allen Wittenauer added a comment - FWIW, I can't duplicate this on Solaris or Mac OS X, both of which are based upon BSD sockets. So I'm guessing this is a bug in the Linux TCP/IP stack.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt
        against trunk revision 937577.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt against trunk revision 937577. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/console This message is automatically generated.
        Hide
        Hong Tang added a comment -

        http://www.rampa.sk/static/tcpLoopConnect.html

        This is not a bug, but the usefulness of this feature is certainly questionable.

        Show
        Hong Tang added a comment - http://www.rampa.sk/static/tcpLoopConnect.html This is not a bug, but the usefulness of this feature is certainly questionable.
        Hide
        Hong Tang added a comment -

        Also, it seems somebody experienced this problem on freebsd too. http://osdir.com/ml/freebsd.devel.hackers/2002-05/msg00209.html

        Show
        Hong Tang added a comment - Also, it seems somebody experienced this problem on freebsd too. http://osdir.com/ml/freebsd.devel.hackers/2002-05/msg00209.html
        Hide
        Todd Lipcon added a comment -

        Hudson bot isn't commenting automatically anymore, but results here:
        hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/

        [exec] +1 overall. Here are the results of testing the latest attachment
        [exec] http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt
        [exec] against trunk revision 937577.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 2 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
        [exec]
        [exec] +1 core tests. The patch passed core unit tests.
        [exec]
        [exec] +1 contrib tests. The patch passed contrib unit tests.

        Show
        Todd Lipcon added a comment - Hudson bot isn't commenting automatically anymore, but results here: hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/478/ [exec] +1 overall. Here are the results of testing the latest attachment [exec] http://issues.apache.org/jira/secure/attachment/12442761/hadoop-6722.txt [exec] against trunk revision 937577. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 core tests. The patch passed core unit tests. [exec] [exec] +1 contrib tests. The patch passed contrib unit tests.
        Hide
        Todd Lipcon added a comment -

        btw, Allen: I was able to reproduce on OSX. It just takes a while sometimes.

        Show
        Todd Lipcon added a comment - btw, Allen: I was able to reproduce on OSX. It just takes a while sometimes.
        Hide
        Tom White added a comment -

        +1

        Show
        Tom White added a comment - +1
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Todd!

        Show
        Tom White added a comment - I've just committed this. Thanks Todd!
        Hide
        Suresh Srinivas added a comment -

        Patch for 0.20-security.

        Show
        Suresh Srinivas added a comment - Patch for 0.20-security.
        Hide
        Jitendra Nath Pandey added a comment -

        +1 for 20s patch.

        Show
        Jitendra Nath Pandey added a comment - +1 for 20s patch.
        Hide
        Suresh Srinivas added a comment -

        I commented the patch to 0.20-security.

        Show
        Suresh Srinivas added a comment - I commented the patch to 0.20-security.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development