Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: python
    • Labels:
      None

      Description

      Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.

      1. 528-support-twisted-5.patch
        16 kB
        Esteve Fernandez
      2. 528-support-twisted-4.patch
        15 kB
        Esteve Fernandez
      3. 528-support-twisted-3.patch
        15 kB
        Esteve Fernandez
      4. 528-support-twisted-2.patch
        8 kB
        Esteve Fernandez
      5. 528-twisted-support.patch
        5 kB
        Esteve Fernandez

        Issue Links

          Activity

          Esteve Fernandez created issue -
          Hide
          Esteve Fernandez added a comment -

          This patch adds support for client-side networking with Avro using HTTP as the transport. A patch which adds support for Avro servers will follow.

          The changes in the patch modify the current Requestor, splitting requests in two phases:

          • serializing the request and sendint it
          • waiting for a response

          these changes don't cause any backwards incompatibiliy, but introduce a new class (BaseRequestor) which delegates to child classes the responsibility of implementing the BaseRequestor#issue_request method

          Show
          Esteve Fernandez added a comment - This patch adds support for client-side networking with Avro using HTTP as the transport. A patch which adds support for Avro servers will follow. The changes in the patch modify the current Requestor, splitting requests in two phases: serializing the request and sendint it waiting for a response these changes don't cause any backwards incompatibiliy, but introduce a new class (BaseRequestor) which delegates to child classes the responsibility of implementing the BaseRequestor#issue_request method
          Esteve Fernandez made changes -
          Field Original Value New Value
          Attachment 528-twisted-support.patch [ 12442778 ]
          Hide
          Esteve Fernandez added a comment -

          This version of the patch adds a streaming producer for issuing requests (http://twistedmatrix.com/documents/current/api/twisted.web.iweb.IBodyProducer.html) and a server-side implementation of the Avro HTTP transport. The latter is implemented as a Twisted.web resource that can be placed in any segment of a URL hierarhcy, supporting multiple Avro protocols to be served by the same server.

          Show
          Esteve Fernandez added a comment - This version of the patch adds a streaming producer for issuing requests ( http://twistedmatrix.com/documents/current/api/twisted.web.iweb.IBodyProducer.html ) and a server-side implementation of the Avro HTTP transport. The latter is implemented as a Twisted.web resource that can be placed in any segment of a URL hierarhcy, supporting multiple Avro protocols to be served by the same server.
          Esteve Fernandez made changes -
          Attachment 528-support-twisted-2.patch [ 12442847 ]
          Esteve Fernandez made changes -
          Component/s python [ 12312781 ]
          Hide
          Esteve Fernandez added a comment -

          This patch includes a full stream-based client (both for incoming and for outgoing data) and two examples of a client and a server using the Twisted portion.

          Unfortunately, the Twisted.web server doesn't support streams, we're working on it, though.

          Show
          Esteve Fernandez added a comment - This patch includes a full stream-based client (both for incoming and for outgoing data) and two examples of a client and a server using the Twisted portion. Unfortunately, the Twisted.web server doesn't support streams, we're working on it, though.
          Esteve Fernandez made changes -
          Attachment 528-support-twisted-3.patch [ 12442871 ]
          Esteve Fernandez made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Esteve Fernandez added a comment -

          This version corrects an error in the way the length attribute of the RequestStreamingProducer is calculated. It can also be set to UNKNOWN_LENGTH, but it's advised that if the body length can be calculated, it should be set.

          However, if there's a way in Avro to pass an object that streams the message, instead of passing the entire message, it'd be possible to use the RequestStreamingProducer without knowing the message length beforehand.

          Show
          Esteve Fernandez added a comment - This version corrects an error in the way the length attribute of the RequestStreamingProducer is calculated. It can also be set to UNKNOWN_LENGTH, but it's advised that if the body length can be calculated, it should be set. However, if there's a way in Avro to pass an object that streams the message, instead of passing the entire message, it'd be possible to use the RequestStreamingProducer without knowing the message length beforehand.
          Esteve Fernandez made changes -
          Attachment 528-support-twisted-4.patch [ 12442954 ]
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve,

          Thanks for the patches! It's good to see you working with Avro. I have been traveling for the past several days, but I'm going to take a look at these soon.

          One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406, in addition to this issue.

          Regards,
          Jeff

          Show
          Jeff Hammerbacher added a comment - Hey Esteve, Thanks for the patches! It's good to see you working with Avro. I have been traveling for the past several days, but I'm going to take a look at these soon. One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406 , in addition to this issue. Regards, Jeff
          Jeff Hammerbacher made changes -
          Link This issue is related to AVRO-406 [ AVRO-406 ]
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-twisted-4.patch), right?

          Show
          Jeff Hammerbacher added a comment - Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-twisted-4.patch), right?
          Hide
          Esteve Fernandez added a comment -

          Hi Jeff, sorry for the slow reply, I've been sick for the last week (I still am, though, argghh).

          > Thanks for the patches! It's good to see you working with Avro. I have been traveling for the
          > past several days, but I'm going to take a look at these soon.

          Thank you very much for your quick response and kind words

          > One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406,
          > in addition to this issue.

          I barely have any experience with Hadoop, and I'm still learning the intricacies of Avro, but I'll try to contribute to the discussion.

          > Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-
          > twisted-4.patch), right?

          Yes, I started some work on making the examples launchable by twistd (Twisted's launcher, which lets you choose any of the available reactors, among other things), but I couldn't finish it. In any case, if you could have a look at the latest patch (528-support-twisted-4.patch), that would be awesome. I can always create another ticket later and submit a patch for twistd and the examples.

          Thanks a lot!

          Show
          Esteve Fernandez added a comment - Hi Jeff, sorry for the slow reply, I've been sick for the last week (I still am, though, argghh). > Thanks for the patches! It's good to see you working with Avro. I have been traveling for the > past several days, but I'm going to take a look at these soon. Thank you very much for your quick response and kind words > One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406 , > in addition to this issue. I barely have any experience with Hadoop, and I'm still learning the intricacies of Avro, but I'll try to contribute to the discussion. > Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support- > twisted-4.patch), right? Yes, I started some work on making the examples launchable by twistd (Twisted's launcher, which lets you choose any of the available reactors, among other things), but I couldn't finish it. In any case, if you could have a look at the latest patch (528-support-twisted-4.patch), that would be awesome. I can always create another ticket later and submit a patch for twistd and the examples. Thanks a lot!
          Jeff Hammerbacher made changes -
          Link This issue relates to AVRO-539 [ AVRO-539 ]
          Hide
          Esteve Fernandez added a comment -

          Added a license header to lang/py/src/avro/txipc.py and updated the patch against trunk

          Show
          Esteve Fernandez added a comment - Added a license header to lang/py/src/avro/txipc.py and updated the patch against trunk
          Esteve Fernandez made changes -
          Attachment 528-support-twisted-5.patch [ 12449446 ]
          Hide
          Esteve Fernandez added a comment -

          Hi Jeff,

          is there anything I can help with to get this ticket solved? I've added a missing license header and updated the patch to fix some offset errors with patch.

          Thanks.

          Show
          Esteve Fernandez added a comment - Hi Jeff, is there anything I can help with to get this ticket solved? I've added a missing license header and updated the patch to fix some offset errors with patch. Thanks.
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve,

          Sorry, I just haven't been able to make reviewing this patch a priority. Now that the 1.4.0 release is imminent, I'll mark this as a blocker and be sure to get it reviewed.

          Thanks,
          Jeff

          Show
          Jeff Hammerbacher added a comment - Hey Esteve, Sorry, I just haven't been able to make reviewing this patch a priority. Now that the 1.4.0 release is imminent, I'll mark this as a blocker and be sure to get it reviewed. Thanks, Jeff
          Jeff Hammerbacher made changes -
          Fix Version/s 1.4.0 [ 12314789 ]
          Hide
          Doug Cutting added a comment -

          Jeff, have you been able to look at this? It's the most voted-on issue. Thanks!

          Show
          Doug Cutting added a comment - Jeff, have you been able to look at this? It's the most voted-on issue. Thanks!
          Hide
          Doug Cutting added a comment -

          This patch applies and passes tests. I will commit it unless someone objects.

          Show
          Doug Cutting added a comment - This patch applies and passes tests. I will commit it unless someone objects.
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Esteve!

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Esteve!
          Doug Cutting made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Assignee Esteve Fernandez [ esteve ]
          Resolution Fixed [ 1 ]
          Doug Cutting made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to AVRO-721 [ AVRO-721 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          1d 1h 2m 1 Esteve Fernandez 26/Apr/10 18:32
          Patch Available Patch Available Resolved Resolved
          122d 3h 44m 1 Doug Cutting 26/Aug/10 22:17
          Resolved Resolved Closed Closed
          12d 23h 50m 1 Doug Cutting 08/Sep/10 22:07

            People

            • Assignee:
              Esteve Fernandez
              Reporter:
              Esteve Fernandez
            • Votes:
              7 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development