Avro
  1. Avro
  2. AVRO-323

Support raw sockets as a transport in avroj

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    1. AVRO-323.patch
      7 kB
      Doug Cutting
    2. AVRO-323.patch
      2 kB
      Doug Cutting

      Activity

      Hide
      Doug Cutting added a comment -

      Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.

      Show
      Doug Cutting added a comment - Here's an early-version of an untested patch that permits one to specify the transport on the command line to rpcreceive. Use 'socket' to get the socket transport.
      Hide
      Jeff Hammerbacher added a comment -

      Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.

      Show
      Jeff Hammerbacher added a comment - Patch applies from inside of lang/java: is that the new intended behavior for patches post-163? If so, we should update the "How to Contribute" wiki page.
      Hide
      Jeff Hammerbacher added a comment -

      Getting an error when I try this out:

      java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{"namespace":"example.proto","protocol":"Mail","types":[{"name":"Message","type":"record","fields":[{"name":"to","type":"string"},{"name":"from","type":"string"},{"name":"body","type":"string"}]}],"messages":{"send":{"request":[{"name":"message","type":"Message"}],"response":"string"},"replay":{"request":[],"response":"string"}}}' 9090 send '{"message": {"to":"avro", "from":"jeff", "body":"hey"}}' socket
      Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      
      Show
      Jeff Hammerbacher added a comment - Getting an error when I try this out: java -jar build/avro-tools-1.2.0-dev.jar rpcreceive '{ "namespace" : "example.proto" , "protocol" : "Mail" , "types" :[{ "name" : "Message" , "type" : "record" , "fields" :[{ "name" : "to" , "type" : "string" },{ "name" : "from" , "type" : "string" },{ "name" : "body" , "type" : "string" }]}], "messages" :{ "send" :{ "request" :[{ "name" : "message" , "type" : "Message" }], "response" : "string" }, "replay" :{ "request" :[], "response" : "string" }}}' 9090 send '{ "message" : { "to" : "avro" , "from" : "jeff" , "body" : "hey" }}' socket Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 5,Size: 5
      Hide
      Jeff Hammerbacher added a comment -

      Doug: would be nice to have rpcsend as well.

      Show
      Jeff Hammerbacher added a comment - Doug: would be nice to have rpcsend as well.
      Hide
      Doug Cutting added a comment -

      Here's a version that:

      • includes tests
      • supports "rpcsend" too
      • actually works!
      Show
      Doug Cutting added a comment - Here's a version that: includes tests supports "rpcsend" too actually works!
      Hide
      Jeff Hammerbacher added a comment -

      In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.

      Show
      Jeff Hammerbacher added a comment - In rpcsend with the socket transport, are you opening the socket with the SO_REUSEADDR option? I'm getting "address already in use" errors after killing the client for a few seconds afterwards. Debugging is easier with SO_REUSEADDR.
      Hide
      Jeff Hammerbacher added a comment -

      When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so:

      19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090
      Exception in thread "main" java.io.IOException: Broken pipe
      	at sun.nio.ch.FileDispatcher.write0(Native Method)
      	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
      	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
      	at sun.nio.ch.IOUtil.write(IOUtil.java:75)
      	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
      	at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78)
      	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33)
      	at org.apache.avro.ipc.Requestor.request(Requestor.java:110)
      	at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80)
      	at org.apache.avro.tool.Main.run(Main.java:73)
      	at org.apache.avro.tool.Main.main(Main.java:62)
      
      Show
      Jeff Hammerbacher added a comment - When debugging rpcsend, I can see that my Python server is receiving the initial client request, parsing the Handshake properly, and noting that it doesn't know the client hash. It's then returning a handshake response and closing the socket. The Java client doesn't know that the socket has been closed on it, so it tries to write its more verbose handshake request, but gets a broken pipe, like so: 19 [main] INFO org.apache.avro.ipc.SocketTransceiver - open to localhost/127.0.0.1:9090 Exception in thread "main" java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.avro.ipc.SocketTransceiver.writeBuffers(SocketTransceiver.java:78) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:33) at org.apache.avro.ipc.Requestor.request(Requestor.java:110) at org.apache.avro.tool.RpcSendTool.run(RpcSendTool.java:80) at org.apache.avro.tool.Main.run(Main.java:73) at org.apache.avro.tool.Main.main(Main.java:62)
      Hide
      Jeff Hammerbacher added a comment -

      Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?

      Show
      Jeff Hammerbacher added a comment - Philip, Doug: Python's out-of-the-box TCPServer implementation will call close() on the socket for each message it handles. There are three lines of logic in the Python client to handle this case (get a new socket if the server has hung up). I'm not sure what the right design decision is here: require all clients to handle the case of a server hanging up mid-handshake, or require all servers to leave the connection open. I think the former makes more sense than the latter, but you guys probably have better taste than me in this regard. Thoughts?
      Hide
      Philip Zeyliger added a comment -

      +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.

      Show
      Philip Zeyliger added a comment - +1 to the patch. In TestRpcReceiveAndSendTools, can you move the JavaDoc, I think the javadoc should be attached to the serveAndSend(protocol) method, instead of one of the test methods.
      Hide
      Doug Cutting added a comment -

      We should not further embrace a raw socket protocol until it's fully specified.

      Show
      Doug Cutting added a comment - We should not further embrace a raw socket protocol until it's fully specified.

        People

        • Assignee:
          Doug Cutting
          Reporter:
          Jeff Hammerbacher
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development