Kafka
  1. Kafka
  2. KAFKA-419

Updated PHP client library to support kafka 0.7+

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.7, 0.8.0
    • Fix Version/s: None
    • Component/s: clients
    • Labels:
    • Environment:
      N/A

      Description

      I updated the PHP client library to fully work with the kafka code in trunk, with the following improvements:

      • added support for GZIP compression (both in the producer and the consumer);
      • implemented stream-based iteration of messages (rather than loading the entire response in memory);
      • added base Kafka_Exception class (extending RuntimeException) and some child classes to catch specific errors;
      • added validation and error checking in several parts of the code (like CRC validation of messages, stream connection/EOF checks, response code verification);
      • added example producer for compressed messages.

      The code is fully tested (vast improvements to the existing test suite are provided with the patch), and already incorporates a proper fix for KAFKA-319 (which can be rejected).
      The API hasn't changed and is 100% backwards compatible.

      1. phpclient.patch
        43 kB
        Lorenzo Alberton
      2. phpclient.new.patch
        126 kB
        Lorenzo Alberton

        Issue Links

          Activity

          Hide
          Lorenzo Alberton added a comment -

          Patch to update the PHP client library to fully work with the kafka code in trunk

          Show
          Lorenzo Alberton added a comment - Patch to update the PHP client library to fully work with the kafka code in trunk
          Hide
          Neha Narkhede added a comment -

          Can't really see the patch here. Could you please upload it using the 'Attach files' option ?

          Show
          Neha Narkhede added a comment - Can't really see the patch here. Could you please upload it using the 'Attach files' option ?
          Hide
          Lorenzo Alberton added a comment -

          Uploading patch again.

          Show
          Lorenzo Alberton added a comment - Uploading patch again.
          Hide
          Lorenzo Alberton added a comment -

          New patch, supercedes phpclient.patch

          Show
          Lorenzo Alberton added a comment - New patch, supercedes phpclient.patch
          Hide
          Lorenzo Alberton added a comment -

          I updated the PHP client library again to support additional functionality and make it more robust.
          The new patch (phpclient.new.patch) completely replaces the old one (phpclient.patch).

          List of improvements on the previous patch:

          • completely refactored socket handling, to be more robust, with better error checking and handling of edge-cases.
          • added support for 64bit offsets
          • better checks for responses from Kafka (fixed connection hanging)
          • added Zookeeper-based consumer, with support for multiple consumer groups, and for manual offset commit action (so it's possible to wait for an ACK from the message processor before advancing the offsets), and example code
          • added support for OffsetRequest and getOffsetsBefore() in the SimpleConsumer class to query the state of the queue
          • new, more specific exception classes (socket errors, empty queue, invalid topic, etc.)
          • support for connection timeouts in microseconds
          • vastly improved test suite
          Show
          Lorenzo Alberton added a comment - I updated the PHP client library again to support additional functionality and make it more robust. The new patch (phpclient.new.patch) completely replaces the old one (phpclient.patch). List of improvements on the previous patch: completely refactored socket handling, to be more robust, with better error checking and handling of edge-cases. added support for 64bit offsets better checks for responses from Kafka (fixed connection hanging) added Zookeeper-based consumer, with support for multiple consumer groups, and for manual offset commit action (so it's possible to wait for an ACK from the message processor before advancing the offsets), and example code added support for OffsetRequest and getOffsetsBefore() in the SimpleConsumer class to query the state of the queue new, more specific exception classes (socket errors, empty queue, invalid topic, etc.) support for connection timeouts in microseconds vastly improved test suite
          Hide
          Lorenzo Alberton added a comment -

          Replaces phpclient.patch

          Show
          Lorenzo Alberton added a comment - Replaces phpclient.patch
          Hide
          Colin B. added a comment -

          Old bug. Client libraries, such as this PHP library, are no longer being maintained in the main project.

          Please close. Won't Fix.

          Show
          Colin B. added a comment - Old bug. Client libraries, such as this PHP library, are no longer being maintained in the main project. Please close. Won't Fix.

            People

            • Assignee:
              Unassigned
              Reporter:
              Lorenzo Alberton
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development