Avro
  1. Avro
  2. AVRO-287

Make RPC interop tests work with new Python implementation

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: python
    • Labels:
      None
    1. AVRO-287.patch
      9 kB
      Jeff Hammerbacher
    2. AVRO-287.patch
      9 kB
      Doug Cutting
    3. AVRO-287.patch
      9 kB
      Doug Cutting
    4. AVRO-287.patch
      9 kB
      Jeff Hammerbacher
    5. AVRO-287.patch
      9 kB
      Jeff Hammerbacher
    6. javaserverpyclient
      12 kB
      Jeff Hammerbacher
    7. AVRO-287.patch
      4 kB
      Jeff Hammerbacher
    8. echo.avpr
      0.1 kB
      Eric Evans
    9. HelloClient.java
      0.6 kB
      Eric Evans
    10. HelloServer.java
      0.7 kB
      Eric Evans
    11. hw.py
      0.4 kB
      Eric Evans

      Issue Links

        Activity

        Hide
        Eric Evans added a comment -

        Python clients and Java servers do not seem to interop, the attached trivial example demonstrates this.

        Show
        Eric Evans added a comment - Python clients and Java servers do not seem to interop, the attached trivial example demonstrates this.
        Hide
        Eric Evans added a comment -

        attached correct schema this time

        Show
        Eric Evans added a comment - attached correct schema this time
        Hide
        Jeff Hammerbacher added a comment -

        Quick note to self: avroj's "rpcsend" utility uses an HTTP transport, so it's not going to work with my Python raw sockets server.

        Show
        Jeff Hammerbacher added a comment - Quick note to self: avroj's "rpcsend" utility uses an HTTP transport, so it's not going to work with my Python raw sockets server.
        Hide
        Jeff Hammerbacher added a comment -

        Here's a patch that implements "rpcsend" and "rpcreceive" for tool.py. Note that I can get Python clients and servers talking to each other just fine, and I can get a Java client talking to a Python server just fine. The Python client can send a message to a Java server, and the Java server happily replies and shuts down, but the Python client, when it calls getresponse(), throws a httplib.ResponseNotReady exception, which is extraordinarily frustrating. I blame Java, of course. I spent a few hours looking at the source but couldn't see anything obviously different between the Python and Java clients.

        Philip, given your knowledge of both languages, could you help me debug?

        Show
        Jeff Hammerbacher added a comment - Here's a patch that implements "rpcsend" and "rpcreceive" for tool.py. Note that I can get Python clients and servers talking to each other just fine, and I can get a Java client talking to a Python server just fine. The Python client can send a message to a Java server, and the Java server happily replies and shuts down, but the Python client, when it calls getresponse(), throws a httplib.ResponseNotReady exception, which is extraordinarily frustrating. I blame Java, of course. I spent a few hours looking at the source but couldn't see anything obviously different between the Python and Java clients. Philip, given your knowledge of both languages, could you help me debug?
        Hide
        Jeff Hammerbacher added a comment -

        Also note: I'm aware that the tool.py code is fugly; broken windows, etc. Just wanted to get things running tonight.

        Show
        Jeff Hammerbacher added a comment - Also note: I'm aware that the tool.py code is fugly; broken windows, etc. Just wanted to get things running tonight.
        Hide
        Philip Zeyliger added a comment -

        Jeff,

        I bet what's happening is that the Java server is shutting down (and closing its connections) before the python has had a chance to slurp everything. See my comment in https://issues.apache.org/jira/browse/AVRO-321?focusedCommentId=12829405&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12829405 .

        – Philip

        Show
        Philip Zeyliger added a comment - Jeff, I bet what's happening is that the Java server is shutting down (and closing its connections) before the python has had a chance to slurp everything. See my comment in https://issues.apache.org/jira/browse/AVRO-321?focusedCommentId=12829405&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12829405 . – Philip
        Hide
        Jeff Hammerbacher added a comment -

        Gah. What's happening here is that httplib.HTTPConnection does not support multiple subsequent requests over a single connection. I can see a number of solutions here but nothing elegant. Will address this weekend--I'm exhausted from debugging today and need to do some paper reviews.

        Show
        Jeff Hammerbacher added a comment - Gah. What's happening here is that httplib.HTTPConnection does not support multiple subsequent requests over a single connection. I can see a number of solutions here but nothing elegant. Will address this weekend--I'm exhausted from debugging today and need to do some paper reviews.
        Hide
        Doug Cutting added a comment -

        > httplib.HTTPConnection does not support multiple subsequent requests over a single connection.

        That shouldn't cause problems. The handshake need not be done over the same connection as the subsequent request. Each are distinct exchanges at the transport layer that require no shared state.

        Show
        Doug Cutting added a comment - > httplib.HTTPConnection does not support multiple subsequent requests over a single connection. That shouldn't cause problems. The handshake need not be done over the same connection as the subsequent request. Each are distinct exchanges at the transport layer that require no shared state.
        Hide
        Jeff Hammerbacher added a comment -

        The current bash script which implements the interop tests starts the server as a background process ,and redirects stdout to a file to capture the port number of the server. Unfortunately, I have not been able to figure out a way for a Python script to capture what is writted to stdout when it is started as a background process. Stderr, on the other hand, works great. Can we change the mechanism for capturing the server's port?

        Show
        Jeff Hammerbacher added a comment - The current bash script which implements the interop tests starts the server as a background process ,and redirects stdout to a file to capture the port number of the server. Unfortunately, I have not been able to figure out a way for a Python script to capture what is writted to stdout when it is started as a background process. Stderr, on the other hand, works great. Can we change the mechanism for capturing the server's port?
        Hide
        Chad Harrington added a comment -

        Hi Jeff,
        I don't know if this helps you or not, but this is the type of code I use for capturing stdout and stderr:
        http://gist.github.com/298407

        If you have a different problem not addressed by this snippet, let me know.

        Thanks for your work on this,

        Chad

        Show
        Chad Harrington added a comment - Hi Jeff, I don't know if this helps you or not, but this is the type of code I use for capturing stdout and stderr: http://gist.github.com/298407 If you have a different problem not addressed by this snippet, let me know. Thanks for your work on this, Chad
        Hide
        Doug Cutting added a comment -

        > I have not been able to figure out a way for a Python script to capture what is writted to stdout when it is started as a background process.

        I'm confused as to why you'd need to do this. We drive the interop tests with a shell script that already does this, so Python doesn't need to.

        That said, I'm fine with using standard error. I initially tried using standard error and grepping for a line starting with "Port:", but I could not get this to work in bash. So, if you can get the bash script working with stderr, given the fact that Java prints other stuff there too, then please feel free to change the line in RpcRecieveTool that logs the port to use stderr and to update the bash script.

        Show
        Doug Cutting added a comment - > I have not been able to figure out a way for a Python script to capture what is writted to stdout when it is started as a background process. I'm confused as to why you'd need to do this. We drive the interop tests with a shell script that already does this, so Python doesn't need to. That said, I'm fine with using standard error. I initially tried using standard error and grepping for a line starting with "Port:", but I could not get this to work in bash. So, if you can get the bash script working with stderr, given the fact that Java prints other stuff there too, then please feel free to change the line in RpcRecieveTool that logs the port to use stderr and to update the bash script.
        Hide
        Jeff Hammerbacher added a comment -

        Chad: I'm quite familiar with popen2, subprocess, command, and friends. Unfortunately these modules are unrelated to this ticket.

        Doug: I think you've misunderstood my problem. I need to capture stdout because that's the methodology you've used in the scripts to get the port name!

        e.g.

        $ python -c "print 'hey'" > blah2.txt
        $ cat blah2.txt 
        hey
        {/code}
        
        

        $ python -c "print 'hey'" & > blah2.txt
        $ cat blah2.txt

        {/code}
        Show
        Jeff Hammerbacher added a comment - Chad: I'm quite familiar with popen2, subprocess, command, and friends. Unfortunately these modules are unrelated to this ticket. Doug: I think you've misunderstood my problem. I need to capture stdout because that's the methodology you've used in the scripts to get the port name! e.g. $ python -c "print 'hey'" > blah2.txt $ cat blah2.txt hey {/code} $ python -c "print 'hey'" & > blah2.txt $ cat blah2.txt {/code}
        Hide
        Jeff Hammerbacher added a comment -

        Yikes, sorry for that poor formatting. No editing possible to correct. The main point is that Python seems to react differently to running in the background than Java. The Java server writes to stdout while running in the background and it's captured in the output file just fine; the Python server tries to do the same thing but the output is not captured.

        Show
        Jeff Hammerbacher added a comment - Yikes, sorry for that poor formatting. No editing possible to correct. The main point is that Python seems to react differently to running in the background than Java. The Java server writes to stdout while running in the background and it's captured in the output file just fine; the Python server tries to do the same thing but the output is not captured.
        Hide
        Doug Cutting added a comment -

        The following works for me:

        python -c "print 'hey'" > blah2.txt &
        cat blah2.txt

        and that's pretty much what the script does.

        Show
        Doug Cutting added a comment - The following works for me: python -c "print 'hey'" > blah2.txt & cat blah2.txt and that's pretty much what the script does.
        Hide
        Jeff Hammerbacher added a comment -

        So I now have the Java client talking happily to the Python server after much debugging (and flushing of stdout and subclassing of HTTPServer). I have the RPC interop bash script running for every combination of Java and Python except the Python client talking to the Java server. This bug is a fun one, as have been the previous two.

        When the Java server is run as a foreground or background process in a separate terminal from the Python client, everything is fine. However, when the Java server is run as a background process in the same terminal as the Python client, the first handshake happens fine, but the server sends the client a block with a correct block length (as if sending the correct response) but empty block content (mysteriously). I've used Wireshark to capture the exchange between the Java server and the Python client and attached it to this issue.

        Any guidance on how to debug the Java server to figure out how the content of the second message is not hitting the wire would be much appreciated.

        Show
        Jeff Hammerbacher added a comment - So I now have the Java client talking happily to the Python server after much debugging (and flushing of stdout and subclassing of HTTPServer). I have the RPC interop bash script running for every combination of Java and Python except the Python client talking to the Java server. This bug is a fun one, as have been the previous two. When the Java server is run as a foreground or background process in a separate terminal from the Python client, everything is fine. However, when the Java server is run as a background process in the same terminal as the Python client, the first handshake happens fine, but the server sends the client a block with a correct block length (as if sending the correct response) but empty block content (mysteriously). I've used Wireshark to capture the exchange between the Java server and the Python client and attached it to this issue. Any guidance on how to debug the Java server to figure out how the content of the second message is not hitting the wire would be much appreciated.
        Hide
        Jeff Hammerbacher added a comment -

        Here's a patch that will reproduce the problem described above.

        Show
        Jeff Hammerbacher added a comment - Here's a patch that will reproduce the problem described above.
        Hide
        Doug Cutting added a comment -

        This patch does not invoke python correctly, so I can't yet see the problem you describe. I get:

        TEST: share/test/interop/rpc/add/onePlusOne
        Traceback (most recent call last):
        File "lang/py/src/avro/tool.py", line 26, in <module>
        from avro import io
        ImportError: No module named avro

        Show
        Doug Cutting added a comment - This patch does not invoke python correctly, so I can't yet see the problem you describe. I get: TEST: share/test/interop/rpc/add/onePlusOne Traceback (most recent call last): File "lang/py/src/avro/tool.py", line 26, in <module> from avro import io ImportError: No module named avro
        Hide
        Jeff Hammerbacher added a comment -

        Try with this patch? A bit hacky, but the cleanest of the solutions I considered.

        Show
        Jeff Hammerbacher added a comment - Try with this patch? A bit hacky, but the cleanest of the solutions I considered.
        Hide
        Jeff Hammerbacher added a comment -

        Anyone with Java skills who has time to debug the Java server side of this exchange would be very handy right now. I am blocked on this issue by Java knowledge.

        Show
        Jeff Hammerbacher added a comment - Anyone with Java skills who has time to debug the Java server side of this exchange would be very handy right now. I am blocked on this issue by Java knowledge.
        Hide
        Eric Evans added a comment -

        I don't know about the interop tests, and I can't really speak to the hackiness since I haven't spent much time looking at the patch or the code it applies to, but... I gave the most recent patch a quick test and I can now use a python client against a java server (cassandra trunk specifically).

        Show
        Eric Evans added a comment - I don't know about the interop tests, and I can't really speak to the hackiness since I haven't spent much time looking at the patch or the code it applies to, but... I gave the most recent patch a quick test and I can now use a python client against a java server (cassandra trunk specifically).
        Hide
        Doug Cutting added a comment -

        I had to make an adjustment to the way PYTHONPATH was set to make bash happy, but rpc interop tests then pass with this patch for me on Linux. +1

        Show
        Doug Cutting added a comment - I had to make an adjustment to the way PYTHONPATH was set to make bash happy, but rpc interop tests then pass with this patch for me on Linux. +1
        Hide
        Jeff Hammerbacher added a comment -

        Hey Doug,

        What kind of adjustments did you make?

        Thanks,
        Jeff

        Show
        Jeff Hammerbacher added a comment - Hey Doug, What kind of adjustments did you make? Thanks, Jeff
        Hide
        Doug Cutting added a comment -

        Jeff, I attached the version that worked for me when I added that comment.

        Show
        Doug Cutting added a comment - Jeff, I attached the version that worked for me when I added that comment.
        Hide
        Jeff Hammerbacher added a comment -

        Doug,

        I should have been more clear: your patch is exactly the same as my previous patch, according to "diff". Could you please describe in words what changes you made?

        Thanks,
        Jeff

        Show
        Jeff Hammerbacher added a comment - Doug, I should have been more clear: your patch is exactly the same as my previous patch, according to "diff". Could you please describe in words what changes you made? Thanks, Jeff
        Hide
        Doug Cutting added a comment -

        Oops. Seems I attached the wrong patch. Try again.

        Show
        Doug Cutting added a comment - Oops. Seems I attached the wrong patch. Try again.
        Hide
        Jeff Hammerbacher added a comment -

        Hey Doug, it's helpful if you describe your changes in words. It looks like you're just exporting the PYTHONPATH environment variable instead of setting it explicitly in the call to the Python client and server. I am fine with either method; did the previous method not work for you? If not, how did it fail?

        Show
        Jeff Hammerbacher added a comment - Hey Doug, it's helpful if you describe your changes in words. It looks like you're just exporting the PYTHONPATH environment variable instead of setting it explicitly in the call to the Python client and server. I am fine with either method; did the previous method not work for you? If not, how did it fail?
        Hide
        Doug Cutting added a comment -

        Your patch failed for me with:

        share/test/interop/bin/test_rpc_interop.sh: line 60: PYTHONPATH=:lang/py/src: No such file or directory

        Can we just commit this now, or are you still hoping to get it to work on MacOS?

        Show
        Doug Cutting added a comment - Your patch failed for me with: share/test/interop/bin/test_rpc_interop.sh: line 60: PYTHONPATH=:lang/py/src: No such file or directory Can we just commit this now, or are you still hoping to get it to work on MacOS?
        Hide
        Jeff Hammerbacher added a comment -

        Adopted Doug's style but appending to, rather than overwriting, the old PYTHONPATH. Works on Mac OS X, mysteriously.

        Show
        Jeff Hammerbacher added a comment - Adopted Doug's style but appending to, rather than overwriting, the old PYTHONPATH. Works on Mac OS X, mysteriously.
        Hide
        Jeff Hammerbacher added a comment -

        Committed revision 910347 and revision 910351.

        Show
        Jeff Hammerbacher added a comment - Committed revision 910347 and revision 910351.

          People

          • Assignee:
            Jeff Hammerbacher
            Reporter:
            Jeff Hammerbacher
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development