Hadoop Common
  1. Hadoop Common
  2. HADOOP-7729

Send back valid HTTP response if user hits IPC port with HTTP GET

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Often, I've seen users get confused between the IPC ports and HTTP ports for a daemon. It would be easy for us to detect when an HTTP GET request hits an IPC port, and instead of sending back garbage, we can send back a valid HTTP response explaining their mistake.

        Activity

        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12498334/hadoop-7729.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in .

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/280//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/280//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498334/hadoop-7729.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/280//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/280//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Patch looks fine to me, Todd. One small question: can we be guaranteed that a legitimate IPC call will not happen to start with the same bytes that correspond to "GET "?

        +1 pending an answer to that question.

        Show
        Aaron T. Myers added a comment - Patch looks fine to me, Todd. One small question: can we be guaranteed that a legitimate IPC call will not happen to start with the same bytes that correspond to "GET " ? +1 pending an answer to that question.
        Hide
        Todd Lipcon added a comment -

        yes, legit Hadoop RPC calls have the header "hrpc" on them (followed by a version number)

        Show
        Todd Lipcon added a comment - yes, legit Hadoop RPC calls have the header "hrpc" on them (followed by a version number)
        Hide
        Aaron T. Myers added a comment -

        Thanks for the prompt response. Commit away.

        Show
        Aaron T. Myers added a comment - Thanks for the prompt response. Commit away.
        Hide
        Steve Loughran added a comment -

        Todd, If there's one thing I'd change, it'd be create a wiki page talking about this problem and include the URL in the response -it's what HADOOP-7469 did for connectivity issues. The people getting confused this way are often new to Hadoop, and a wiki page can give them more detail on what they need to look for instead, including the various configuration files that will tell them what port the HTTP server is on.

        Now, if you are really devious and the Server instance was told the port the HTTP server for that service was up on, it could give the user the right URL (ignoring multiple hostname issues). I wouldn't try and 30x back as that would only create more confusion.

        Incidentally, what if the client is an HTTP1.0 browser? Shouldn't the HTTP version be grabbed from the incoming request? Or is that getting too close to building a mini-web-server?

        Show
        Steve Loughran added a comment - Todd, If there's one thing I'd change, it'd be create a wiki page talking about this problem and include the URL in the response -it's what HADOOP-7469 did for connectivity issues. The people getting confused this way are often new to Hadoop, and a wiki page can give them more detail on what they need to look for instead, including the various configuration files that will tell them what port the HTTP server is on. Now, if you are really devious and the Server instance was told the port the HTTP server for that service was up on, it could give the user the right URL (ignoring multiple hostname issues). I wouldn't try and 30x back as that would only create more confusion. Incidentally, what if the client is an HTTP1.0 browser? Shouldn't the HTTP version be grabbed from the incoming request? Or is that getting too close to building a mini-web-server?
        Hide
        Todd Lipcon added a comment -

        Hi Steve. I agree that we could do even more to help new users. But as some silly philosopher once said, "perfect is the enemy of the good". Another philosopher said something like "patches welcome" Mind if I commit as is and we can improve it further with another JIRA if you want?

        In response to your particular points:

        Now, if you are really devious and the Server instance was told the port the HTTP server for that service was up on

        This is problematic since some servers have lots of different HTTP endpoints, and also because the IPC portion of Hadoop is nicely divorced from the HTTP portion of Hadoop right now. Adding a dependence between them seems messy.

        Incidentally, what if the client is an HTTP1.0 browser? Shouldn't the HTTP version be grabbed from the incoming request? Or is that getting too close to building a mini-web-server?

        I tested the patch with curl and google chrome. Both seemed to work fine. Given that HTTP 1.1 was introduced in 1999, I don't know if any HTTP 1.0 browsers are even out there anymore. Again, if someone has an incompatible browser, they're still no worse off with the patch than they were before.

        Show
        Todd Lipcon added a comment - Hi Steve. I agree that we could do even more to help new users. But as some silly philosopher once said, "perfect is the enemy of the good". Another philosopher said something like "patches welcome" Mind if I commit as is and we can improve it further with another JIRA if you want? In response to your particular points: Now, if you are really devious and the Server instance was told the port the HTTP server for that service was up on This is problematic since some servers have lots of different HTTP endpoints, and also because the IPC portion of Hadoop is nicely divorced from the HTTP portion of Hadoop right now. Adding a dependence between them seems messy. Incidentally, what if the client is an HTTP1.0 browser? Shouldn't the HTTP version be grabbed from the incoming request? Or is that getting too close to building a mini-web-server? I tested the patch with curl and google chrome. Both seemed to work fine. Given that HTTP 1.1 was introduced in 1999, I don't know if any HTTP 1.0 browsers are even out there anymore. Again, if someone has an incompatible browser, they're still no worse off with the patch than they were before.
        Hide
        Suresh Srinivas added a comment -

        +1 for the patch.

        Show
        Suresh Srinivas added a comment - +1 for the patch.
        Hide
        Steve Loughran added a comment -

        +1
        If HTTP/1.0 compatibility isn't an issue, ship it. The only people who would have a problem are those doing telnet and issuing GET commands by hand, but they aren't the users we care about here. If you can do that, you know how to do a netstat -a -p tcp and work out for themselves what they are doing wrong.

        Of course, any protocol HTTP included that doesn't include some bit of response from the server is wasting some of the SYN-ACK packet used in setting up the TCP connection -in an ideal world that is where all protocols ought to publish some identifier and version number. If the current TCP-based protocols don't do that, it's not easily fixed.

        Show
        Steve Loughran added a comment - +1 If HTTP/1.0 compatibility isn't an issue, ship it. The only people who would have a problem are those doing telnet and issuing GET commands by hand, but they aren't the users we care about here. If you can do that, you know how to do a netstat -a -p tcp and work out for themselves what they are doing wrong. Of course, any protocol HTTP included that doesn't include some bit of response from the server is wasting some of the SYN-ACK packet used in setting up the TCP connection -in an ideal world that is where all protocols ought to publish some identifier and version number. If the current TCP-based protocols don't do that, it's not easily fixed.
        Hide
        Todd Lipcon added a comment -

        Fixed only in trunk. Would be happy to apply to 23 as well if people think it's worth doing.

        Show
        Todd Lipcon added a comment - Fixed only in trunk. Would be happy to apply to 23 as well if people think it's worth doing.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1084 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1084/)
        HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1084 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1084/ ) HADOOP-7729 . Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1162/)
        HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1162 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1162/ ) HADOOP-7729 . Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1103 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1103/)
        HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1103 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1103/ ) HADOOP-7729 . Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #861 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/861/)
        HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #861 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/861/ ) HADOOP-7729 . Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #831 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/831/)
        HADOOP-7729. Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon.

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #831 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/831/ ) HADOOP-7729 . Send back valid HTTP response if user hits IPC port with HTTP GET. Contributed by Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183512 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #650 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/650/)
        HADOOP-7729. Merge r1183512 from trunk to 0.23 (Revision 1298085)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #650 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/650/ ) HADOOP-7729 . Merge r1183512 from trunk to 0.23 (Revision 1298085) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #659 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/659/)
        HADOOP-7729. Merge r1183512 from trunk to 0.23 (Revision 1298085)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #659 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/659/ ) HADOOP-7729 . Merge r1183512 from trunk to 0.23 (Revision 1298085) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #660 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/660/)
        HADOOP-7729. Merge r1183512 from trunk to 0.23 (Revision 1298085)

        Result = SUCCESS
        suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #660 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/660/ ) HADOOP-7729 . Merge r1183512 from trunk to 0.23 (Revision 1298085) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1298085 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development