Kafka
  1. Kafka
  2. KAFKA-160

ZK consumer gets into infinite loop if a message is larger than fetch size

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: core
    • Labels:
      None
    1. KAFKA-160.patch
      2 kB
      Jun Rao
    2. KAFKA-160_v2.patch
      4 kB
      Jun Rao

      Activity

      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Patch Available Patch Available
      27m 49s 1 Jun Rao 18/Oct/11 18:39
      Patch Available Patch Available Resolved Resolved
      1d 6h 14m 1 Jun Rao 20/Oct/11 00:53
      Jun Rao made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Resolution Fixed [ 1 ]
      Hide
      Jun Rao added a comment -

      Committed.

      Show
      Jun Rao added a comment - Committed.
      Hide
      Neha Narkhede added a comment -

      +1 on patch v2

      Show
      Neha Narkhede added a comment - +1 on patch v2
      Jun Rao made changes -
      Attachment KAFKA-160_v2.patch [ 12499702 ]
      Hide
      Jun Rao added a comment -

      Submitted patch v2, adding a unit test that uncovers the problem.

      Show
      Jun Rao added a comment - Submitted patch v2, adding a unit test that uncovers the problem.
      Neha Narkhede made changes -
      Fix Version/s 0.7 [ 12317243 ]
      Priority Major [ 3 ] Blocker [ 1 ]
      Hide
      Jun Rao added a comment -

      I think this was introduced when compression was added. I reviewed the patch, but didn't catch this.

      Show
      Jun Rao added a comment - I think this was introduced when compression was added. I reviewed the patch, but didn't catch this.
      Hide
      Jay Kreps added a comment -

      Hey did I break this?

      Show
      Jay Kreps added a comment - Hey did I break this?
      Jun Rao made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Jun Rao added a comment -

      There were a couple of problems.
      1. We used to throw an exception while iterating messages in a ByteBufferMessageSet, if we can't iterate a single message out of the buffer. This indicates that either the fetch size is too small or if there is corruption in the data. Throwing exception is a good way to notify the consumer that something is wrong. After the compression patch, such an exception is no longer thrown.

      2. The constructor of ByetBufferMessageSet recently added a new parameter in the middle and some of the callers didn't get changed accordingly. So, some input parameters are misaligned.

      Attach a patch that fixes both problems.

      Show
      Jun Rao added a comment - There were a couple of problems. 1. We used to throw an exception while iterating messages in a ByteBufferMessageSet, if we can't iterate a single message out of the buffer. This indicates that either the fetch size is too small or if there is corruption in the data. Throwing exception is a good way to notify the consumer that something is wrong. After the compression patch, such an exception is no longer thrown. 2. The constructor of ByetBufferMessageSet recently added a new parameter in the middle and some of the callers didn't get changed accordingly. So, some input parameters are misaligned. Attach a patch that fixes both problems.
      Jun Rao made changes -
      Field Original Value New Value
      Attachment KAFKA-160.patch [ 12499567 ]
      Jun Rao created issue -

        People

        • Assignee:
          Jun Rao
          Reporter:
          Jun Rao
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development