Avro
  1. Avro
  2. AVRO-1028

IPC transceiver doesn't gracefully handle server connection resets.

    Details

    • Type: Improvement Improvement
    • Status: Reopened
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.2
    • Fix Version/s: None
    • Component/s: python
    • Labels:
      None

      Description

      The current Python HTTPTransceiver class forces users to handle connection resets.

      I've refactored the class using urllib3 and incorporated some features we get for free from said library into the transceiver. Added a test case for test_ipc.py that uses the twisted server implementation.

        Activity

        Hide
        Bo Shi added a comment -

        I am out of the country right now but will take a look tomorrow/Friday when I get back!

        Show
        Bo Shi added a comment - I am out of the country right now but will take a look tomorrow/Friday when I get back!
        Hide
        Doug Cutting added a comment -

        I reverted this.

        Show
        Doug Cutting added a comment - I reverted this.
        Hide
        Miki Tebeka added a comment -

        This imports the local io instead of the build in io module.
        IMO revert the fix and we'll address that after 1.7.

        Show
        Miki Tebeka added a comment - This imports the local io instead of the build in io module. IMO revert the fix and we'll address that after 1.7.
        Hide
        Doug Cutting added a comment -

        The failure is:

        TEST: share/test/interop/rpc/add/onePlusOne
        Traceback (most recent call last):
        File "lang/py/build/src/avro/tool.py", line 28, in <module>
        from avro import ipc
        File "/home/cutting/src/avro/trunk/lang/py/build/src/avro/ipc.py", line 20, in <module>
        from urllib3.connectionpool import HTTPConnectionPool
        File "/usr/local/lib/python2.7/dist-packages/urllib3/_init_.py", line 16, in <module>
        from .connectionpool import (
        File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 42, in <module>
        from .request import RequestMethods
        File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 12, in <module>
        from .filepost import encode_multipart_formdata
        File "/usr/local/lib/python2.7/dist-packages/urllib3/filepost.py", line 15, in <module>
        from io import BytesIO
        ImportError: cannot import name BytesIO

        Show
        Doug Cutting added a comment - The failure is: TEST: share/test/interop/rpc/add/onePlusOne Traceback (most recent call last): File "lang/py/build/src/avro/tool.py", line 28, in <module> from avro import ipc File "/home/cutting/src/avro/trunk/lang/py/build/src/avro/ipc.py", line 20, in <module> from urllib3.connectionpool import HTTPConnectionPool File "/usr/local/lib/python2.7/dist-packages/urllib3/_ init _.py", line 16, in <module> from .connectionpool import ( File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 42, in <module> from .request import RequestMethods File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 12, in <module> from .filepost import encode_multipart_formdata File "/usr/local/lib/python2.7/dist-packages/urllib3/filepost.py", line 15, in <module> from io import BytesIO ImportError: cannot import name BytesIO
        Hide
        Doug Cutting added a comment -

        Turns out this breaks the RPC interop testing.

        To see this most easily, edit share/test/interop/bin/test_rpc_interop.sh to just test python against python:

        -clients=("$java_client" "$py_client" "$ruby_client")
        -servers=("$java_server" "$py_server" "$ruby_server")
        +clients=("$py_client")
        +servers=("$py_server")

        Then run:

        (cd lang/py; ant)
        bash share/test/interop/bin/test_rpc_interop.sh

        This should print RPC INTEROP TESTS PASS and does if you revert this patch but not with this patch.

        I'd like to get 1.7.0 out soon, so if someone doesn't provide a fix for this I'll revert this and we can get it into 1.7.1.

        Show
        Doug Cutting added a comment - Turns out this breaks the RPC interop testing. To see this most easily, edit share/test/interop/bin/test_rpc_interop.sh to just test python against python: -clients=("$java_client" "$py_client" "$ruby_client") -servers=("$java_server" "$py_server" "$ruby_server") +clients=("$py_client") +servers=("$py_server") Then run: (cd lang/py; ant) bash share/test/interop/bin/test_rpc_interop.sh This should print RPC INTEROP TESTS PASS and does if you revert this patch but not with this patch. I'd like to get 1.7.0 out soon, so if someone doesn't provide a fix for this I'll revert this and we can get it into 1.7.1.
        Hide
        Doug Cutting added a comment -

        I committed this. Thanks, Bo!

        Show
        Doug Cutting added a comment - I committed this. Thanks, Bo!
        Hide
        Bo Shi added a comment -

        You're the boss

        Show
        Bo Shi added a comment - You're the boss
        Hide
        Doug Cutting added a comment -

        Patch works for me, but with the addition of urllib3 requirement perhaps we should push this to 1.7.0?

        Show
        Doug Cutting added a comment - Patch works for me, but with the addition of urllib3 requirement perhaps we should push this to 1.7.0?
        Hide
        Bo Shi added a comment -

        Also, note that this patch incorporates redirect functionality as originally requested in

        https://issues.apache.org/jira/browse/AVRO-665

        Show
        Bo Shi added a comment - Also, note that this patch incorporates redirect functionality as originally requested in https://issues.apache.org/jira/browse/AVRO-665
        Hide
        Bo Shi added a comment -

        Currently using multiprocessing.Process to start a test server. I sleep 1 second to wait for the server to start allowing connections - anyone have a more robust solution offhand?

        Show
        Bo Shi added a comment - Currently using multiprocessing.Process to start a test server. I sleep 1 second to wait for the server to start allowing connections - anyone have a more robust solution offhand?

          People

          • Assignee:
            Bo Shi
            Reporter:
            Bo Shi
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development