Avro
  1. Avro
  2. AVRO-323

Support raw sockets as a transport in avroj

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    1. AVRO-323.patch
      2 kB
      Doug Cutting
    2. AVRO-323.patch
      7 kB
      Doug Cutting

      Activity

      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Resolved Resolved
      19d 21h 40m 1 Doug Cutting 02/Feb/10 21:39
      Resolved Resolved Closed Closed
      27d 8m 1 Doug Cutting 01/Mar/10 21:47
      Doug Cutting made changes -
      Status Resolved [ 5 ] Closed [ 6 ]
      Doug Cutting made changes -
      Status Open [ 1 ] Resolved [ 5 ]
      Resolution Won't Fix [ 2 ]
      Hide
      Doug Cutting added a comment -

      We should not further embrace a raw socket protocol until it's fully specified.

      Show
      Doug Cutting added a comment - We should not further embrace a raw socket protocol until it's fully specified.
      Hide
      Philip Zeyliger added a comment -

      +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.

      Show
      Philip Zeyliger added a comment - +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.
      Hide
      Jeff Hammerbacher added a comment -

      Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?

      Show
      Jeff Hammerbacher added a comment - Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?
      Hide
      Jeff Hammerbacher added a comment -

      When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so:

      19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090
      Exception in thread "main" java.io.IOException: Broken pipe
      	at sun.nio.ch.FileDispatcher.write0(Native Method)
      	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
      	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
      	at sun.nio.ch.IOUtil.write(IOUtil.java:75)
      	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
      	at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78)
      	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33)
      	at org.apache.avro.ipc.Requestor.request(Requestor.java:110)
      	at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80)
      	at org.apache.avro.tool.Main.run(Main.java:73)
      	at org.apache.avro.tool.Main.main(Main.java:62)
      
      Show
      Jeff Hammerbacher added a comment - When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so: 19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090 Exception in thread "main" java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33) at org.apache.avro.ipc.Requestor.request(Requestor.java:110) at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80) at org.apache.avro.tool.Main.run(Main.java:73) at org.apache.avro.tool.Main.main(Main.java:62)
      Hide
      Jeff Hammerbacher added a comment -

      In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.

      Show
      Jeff Hammerbacher added a comment - In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.
      Doug Cutting made changes -
      Assignee Doug Cutting [ cutting ]
      Doug Cutting made changes -
      Attachment AVRO-323.patch [ 12430229 ]
      Hide
      Doug Cutting added a comment -

      Here's a version that:

      • includes tests
      • supports "rpcsend" too
      • actually works!
      Show
      Doug Cutting added a comment - Here's a version that: includes tests supports "rpcsend" too actually works!
      Hide
      Jeff Hammerbacher added a comment -

      Doug: would be nice to have rpcsend as well.

      Show
      Jeff Hammerbacher added a comment - Doug: would be nice to have rpcsend as well.
      Hide
      Jeff Hammerbacher added a comment -

      Getting an error when I try this out:

      java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{"namespace":"example.proto","protocol":"Mail","types":[{"name":"Message","type":"record","fields":[{"name":"to","type":"string"},{"name":"from","type":"string"},{"name":"body","type":"string"}]}],"messages":{"send":{"request":[{"name":"message","type":"Message"}],"response":"string"},"replay":{"request":[],"response":"string"}}}' 9090 send '{"message": {"to":"avro", "from":"jeff", "body":"hey"}}' socket
      Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      
      Show
      Jeff Hammerbacher added a comment - Getting an error when I try this out: java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{ "namespace" : "example.proto" , "protocol" : "Mail" , "types" :[{ "name" : "Message" , "type" : "record" , "fields" :[{ "name" : "to" , "type" : "string" },{ "name" : "from" , "type" : "string" },{ "name" : "body" , "type" : "string" }]}], "messages" :{ "send" :{ "request" :[{ "name" : "message" , "type" : "Message" }], "response" : "string" }, "replay" :{ "request" :[], "response" : "string" }}}' 9090 send '{ "message" : { "to" : "avro" , "from" : "jeff" , "body" : "hey" }}' socket Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      Hide
      Jeff Hammerbacher added a comment -

      Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.

      Show
      Jeff Hammerbacher added a comment - Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.
      Doug Cutting made changes -
      Field Original Value New Value
      Attachment AVRO-323.patch [ 12430191 ]
      Hide
      Doug Cutting added a comment -

      Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.

      Show
      Doug Cutting added a comment - Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.
      Jeff Hammerbacher created issue -

        People

        • Assignee:
          Doug Cutting
          Reporter:
          Jeff Hammerbacher
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development