Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-993

Offset Management API is either broken or mis-documented

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.8.1
    • Fix Version/s: None
    • Component/s: network
    • Labels:
      None

      Description

      I am in the process of building a set of Go client bindings for the new 0.8 protocol (https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol). Everything works but the Offset Commit/Fetch APIs. Fetch never returns any data, and trying to Commit results in the broker forcibly disconnecting my client. I have double-checked the bytes on the wire using Wireshark, and my client is obeying the protocol spec.

      After some digging, I found KAFKA-852 which seems related, but I have tried my client against the 0.8 beta, 0.8 branch, and even trunk with the same results.

      When I try and commit, the stack-trace that the broker produces is:
      [2013-07-31 10:34:14,423] ERROR Closing socket for /192.168.12.71 because of error (kafka.network.Processor)
      java.nio.BufferUnderflowException
      at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
      at java.nio.ByteBuffer.get(ByteBuffer.java:675)
      at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:38)
      at kafka.api.UpdateMetadataRequest$$anonfun$readFrom$1.apply(UpdateMetadataRequest.scala:42)
      at kafka.api.UpdateMetadataRequest$$anonfun$readFrom$1.apply(UpdateMetadataRequest.scala:41)
      at scala.collection.immutable.Range$ByOne$class.foreach(Range.scala:282)
      at scala.collection.immutable.Range$$anon$2.foreach(Range.scala:265)
      at kafka.api.UpdateMetadataRequest$.readFrom(UpdateMetadataRequest.scala:41)
      at kafka.api.RequestKeys$$anonfun$7.apply(RequestKeys.scala:42)
      at kafka.api.RequestKeys$$anonfun$7.apply(RequestKeys.scala:42)
      at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:49)
      at kafka.network.Processor.read(SocketServer.scala:345)
      at kafka.network.Processor.run(SocketServer.scala:245)
      at java.lang.Thread.run(Thread.java:680)

      Is this a bug, or is the protocol spec wrong? Also, since I can't seem to find a straight answer anywhere else: is offset fetch/commit expected to be in 0.8, 0.8.1, or some later release?

      Thanks,
      Evan

        Issue Links

          Activity

          Hide
          eapache Evan Huus added a comment -

          The library is now available at https://github.com/Shopify/sarama

          In case somebody wants to add it to the wiki page of client libraries...

          Show
          eapache Evan Huus added a comment - The library is now available at https://github.com/Shopify/sarama In case somebody wants to add it to the wiki page of client libraries...
          Hide
          lanzaa Colin B. added a comment -

          I believe this is more of a documentation bug. In the push to get Kafka 0.8 out I think it was decided to postpone the release of the server side Offset Commit API.

          As such, the OffsetCommit APIs are only available post 0.8.

          Show
          lanzaa Colin B. added a comment - I believe this is more of a documentation bug. In the push to get Kafka 0.8 out I think it was decided to postpone the release of the server side Offset Commit API. As such, the OffsetCommit APIs are only available post 0.8.
          Hide
          eapache Evan Huus added a comment -

          I have updated the protocol wiki to note that these APIs are not available in 0.8. Thanks for the clarification.

          Show
          eapache Evan Huus added a comment - I have updated the protocol wiki to note that these APIs are not available in 0.8. Thanks for the clarification.
          Hide
          lanzaa Colin B. added a comment -

          Thank you for the help!

          Show
          lanzaa Colin B. added a comment - Thank you for the help!

            People

            • Assignee:
              junrao Jun Rao
              Reporter:
              eapache Evan Huus
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development