Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-323

Support raw sockets as a transport in avroj

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    1. AVRO-323.patch
      7 kB
      Doug Cutting
    2. AVRO-323.patch
      2 kB
      Doug Cutting

      Activity

      Hide
      cutting Doug Cutting added a comment -

      Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.

      Show
      cutting Doug Cutting added a comment - Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.
      Hide
      hammer Jeff Hammerbacher added a comment -

      Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.

      Show
      hammer Jeff Hammerbacher added a comment - Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.
      Hide
      hammer Jeff Hammerbacher added a comment -

      Getting an error when I try this out:

      java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{"namespace":"example.proto","protocol":"Mail","types":[{"name":"Message","type":"record","fields":[{"name":"to","type":"string"},{"name":"from","type":"string"},{"name":"body","type":"string"}]}],"messages":{"send":{"request":[{"name":"message","type":"Message"}],"response":"string"},"replay":{"request":[],"response":"string"}}}' 9090 send '{"message": {"to":"avro", "from":"jeff", "body":"hey"}}' socket
      Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      
      Show
      hammer Jeff Hammerbacher added a comment - Getting an error when I try this out: java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{ "namespace" : "example.proto" , "protocol" : "Mail" , "types" :[{ "name" : "Message" , "type" : "record" , "fields" :[{ "name" : "to" , "type" : "string" },{ "name" : "from" , "type" : "string" },{ "name" : "body" , "type" : "string" }]}], "messages" :{ "send" :{ "request" :[{ "name" : "message" , "type" : "Message" }], "response" : "string" }, "replay" :{ "request" :[], "response" : "string" }}}' 9090 send '{ "message" : { "to" : "avro" , "from" : "jeff" , "body" : "hey" }}' socket Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      Hide
      hammer Jeff Hammerbacher added a comment -

      Doug: would be nice to have rpcsend as well.

      Show
      hammer Jeff Hammerbacher added a comment - Doug: would be nice to have rpcsend as well.
      Hide
      cutting Doug Cutting added a comment -

      Here's a version that:

      • includes tests
      • supports "rpcsend" too
      • actually works!
      Show
      cutting Doug Cutting added a comment - Here's a version that: includes tests supports "rpcsend" too actually works!
      Hide
      hammer Jeff Hammerbacher added a comment -

      In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.

      Show
      hammer Jeff Hammerbacher added a comment - In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.
      Hide
      hammer Jeff Hammerbacher added a comment -

      When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so:

      19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090
      Exception in thread "main" java.io.IOException: Broken pipe
      	at sun.nio.ch.FileDispatcher.write0(Native Method)
      	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
      	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
      	at sun.nio.ch.IOUtil.write(IOUtil.java:75)
      	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
      	at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78)
      	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33)
      	at org.apache.avro.ipc.Requestor.request(Requestor.java:110)
      	at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80)
      	at org.apache.avro.tool.Main.run(Main.java:73)
      	at org.apache.avro.tool.Main.main(Main.java:62)
      
      Show
      hammer Jeff Hammerbacher added a comment - When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so: 19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090 Exception in thread "main" java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33) at org.apache.avro.ipc.Requestor.request(Requestor.java:110) at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80) at org.apache.avro.tool.Main.run(Main.java:73) at org.apache.avro.tool.Main.main(Main.java:62)
      Hide
      hammer Jeff Hammerbacher added a comment -

      Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?

      Show
      hammer Jeff Hammerbacher added a comment - Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?
      Hide
      philip Philip Zeyliger added a comment -

      +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.

      Show
      philip Philip Zeyliger added a comment - +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.
      Hide
      cutting Doug Cutting added a comment -

      We should not further embrace a raw socket protocol until it's fully specified.

      Show
      cutting Doug Cutting added a comment - We should not further embrace a raw socket protocol until it's fully specified.

        People

        • Assignee:
          cutting Doug Cutting
          Reporter:
          hammer Jeff Hammerbacher
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development