Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.4.0
    • Component/s: python
    • Labels:
      None

      Description

      Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.

      1. 528-support-twisted-5.patch
        16 kB
        Esteve Fernandez
      2. 528-support-twisted-4.patch
        15 kB
        Esteve Fernandez
      3. 528-support-twisted-3.patch
        15 kB
        Esteve Fernandez
      4. 528-support-twisted-2.patch
        8 kB
        Esteve Fernandez
      5. 528-twisted-support.patch
        5 kB
        Esteve Fernandez

        Issue Links

          Activity

          Hide
          Esteve Fernandez added a comment -

          This patch adds support for client-side networking with Avro using HTTP as the transport. A patch which adds support for Avro servers will follow.

          The changes in the patch modify the current Requestor, splitting requests in two phases:

          • serializing the request and sendint it
          • waiting for a response

          these changes don't cause any backwards incompatibiliy, but introduce a new class (BaseRequestor) which delegates to child classes the responsibility of implementing the BaseRequestor#issue_request method

          Show
          Esteve Fernandez added a comment - This patch adds support for client-side networking with Avro using HTTP as the transport. A patch which adds support for Avro servers will follow. The changes in the patch modify the current Requestor, splitting requests in two phases: serializing the request and sendint it waiting for a response these changes don't cause any backwards incompatibiliy, but introduce a new class (BaseRequestor) which delegates to child classes the responsibility of implementing the BaseRequestor#issue_request method
          Hide
          Esteve Fernandez added a comment -

          This version of the patch adds a streaming producer for issuing requests (http://twistedmatrix.com/documents/current/api/twisted.web.iweb.IBodyProducer.html) and a server-side implementation of the Avro HTTP transport. The latter is implemented as a Twisted.web resource that can be placed in any segment of a URL hierarhcy, supporting multiple Avro protocols to be served by the same server.

          Show
          Esteve Fernandez added a comment - This version of the patch adds a streaming producer for issuing requests ( http://twistedmatrix.com/documents/current/api/twisted.web.iweb.IBodyProducer.html ) and a server-side implementation of the Avro HTTP transport. The latter is implemented as a Twisted.web resource that can be placed in any segment of a URL hierarhcy, supporting multiple Avro protocols to be served by the same server.
          Hide
          Esteve Fernandez added a comment -

          This patch includes a full stream-based client (both for incoming and for outgoing data) and two examples of a client and a server using the Twisted portion.

          Unfortunately, the Twisted.web server doesn't support streams, we're working on it, though.

          Show
          Esteve Fernandez added a comment - This patch includes a full stream-based client (both for incoming and for outgoing data) and two examples of a client and a server using the Twisted portion. Unfortunately, the Twisted.web server doesn't support streams, we're working on it, though.
          Hide
          Esteve Fernandez added a comment -

          This version corrects an error in the way the length attribute of the RequestStreamingProducer is calculated. It can also be set to UNKNOWN_LENGTH, but it's advised that if the body length can be calculated, it should be set.

          However, if there's a way in Avro to pass an object that streams the message, instead of passing the entire message, it'd be possible to use the RequestStreamingProducer without knowing the message length beforehand.

          Show
          Esteve Fernandez added a comment - This version corrects an error in the way the length attribute of the RequestStreamingProducer is calculated. It can also be set to UNKNOWN_LENGTH, but it's advised that if the body length can be calculated, it should be set. However, if there's a way in Avro to pass an object that streams the message, instead of passing the entire message, it'd be possible to use the RequestStreamingProducer without knowing the message length beforehand.
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve,

          Thanks for the patches! It's good to see you working with Avro. I have been traveling for the past several days, but I'm going to take a look at these soon.

          One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406, in addition to this issue.

          Regards,
          Jeff

          Show
          Jeff Hammerbacher added a comment - Hey Esteve, Thanks for the patches! It's good to see you working with Avro. I have been traveling for the past several days, but I'm going to take a look at these soon. One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406 , in addition to this issue. Regards, Jeff
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-twisted-4.patch), right?

          Show
          Jeff Hammerbacher added a comment - Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-twisted-4.patch), right?
          Hide
          Esteve Fernandez added a comment -

          Hi Jeff, sorry for the slow reply, I've been sick for the last week (I still am, though, argghh).

          > Thanks for the patches! It's good to see you working with Avro. I have been traveling for the
          > past several days, but I'm going to take a look at these soon.

          Thank you very much for your quick response and kind words

          > One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406,
          > in addition to this issue.

          I barely have any experience with Hadoop, and I'm still learning the intricacies of Avro, but I'll try to contribute to the discussion.

          > Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-
          > twisted-4.patch), right?

          Yes, I started some work on making the examples launchable by twistd (Twisted's launcher, which lets you choose any of the available reactors, among other things), but I couldn't finish it. In any case, if you could have a look at the latest patch (528-support-twisted-4.patch), that would be awesome. I can always create another ticket later and submit a patch for twistd and the examples.

          Thanks a lot!

          Show
          Esteve Fernandez added a comment - Hi Jeff, sorry for the slow reply, I've been sick for the last week (I still am, though, argghh). > Thanks for the patches! It's good to see you working with Avro. I have been traveling for the > past several days, but I'm going to take a look at these soon. Thank you very much for your quick response and kind words > One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406 , > in addition to this issue. I barely have any experience with Hadoop, and I'm still learning the intricacies of Avro, but I'll try to contribute to the discussion. > Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support- > twisted-4.patch), right? Yes, I started some work on making the examples launchable by twistd (Twisted's launcher, which lets you choose any of the available reactors, among other things), but I couldn't finish it. In any case, if you could have a look at the latest patch (528-support-twisted-4.patch), that would be awesome. I can always create another ticket later and submit a patch for twistd and the examples. Thanks a lot!
          Hide
          Esteve Fernandez added a comment -

          Added a license header to lang/py/src/avro/txipc.py and updated the patch against trunk

          Show
          Esteve Fernandez added a comment - Added a license header to lang/py/src/avro/txipc.py and updated the patch against trunk
          Hide
          Esteve Fernandez added a comment -

          Hi Jeff,

          is there anything I can help with to get this ticket solved? I've added a missing license header and updated the patch to fix some offset errors with patch.

          Thanks.

          Show
          Esteve Fernandez added a comment - Hi Jeff, is there anything I can help with to get this ticket solved? I've added a missing license header and updated the patch to fix some offset errors with patch. Thanks.
          Hide
          Jeff Hammerbacher added a comment -

          Hey Esteve,

          Sorry, I just haven't been able to make reviewing this patch a priority. Now that the 1.4.0 release is imminent, I'll mark this as a blocker and be sure to get it reviewed.

          Thanks,
          Jeff

          Show
          Jeff Hammerbacher added a comment - Hey Esteve, Sorry, I just haven't been able to make reviewing this patch a priority. Now that the 1.4.0 release is imminent, I'll mark this as a blocker and be sure to get it reviewed. Thanks, Jeff
          Hide
          Doug Cutting added a comment -

          Jeff, have you been able to look at this? It's the most voted-on issue. Thanks!

          Show
          Doug Cutting added a comment - Jeff, have you been able to look at this? It's the most voted-on issue. Thanks!
          Hide
          Doug Cutting added a comment -

          This patch applies and passes tests. I will commit it unless someone objects.

          Show
          Doug Cutting added a comment - This patch applies and passes tests. I will commit it unless someone objects.
          Hide
          Doug Cutting added a comment -

          I just committed this. Thanks, Esteve!

          Show
          Doug Cutting added a comment - I just committed this. Thanks, Esteve!

            People

            • Assignee:
              Esteve Fernandez
              Reporter:
              Esteve Fernandez
            • Votes:
              7 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development