Kafka
  1. Kafka
  2. KAFKA-160

ZK consumer gets into infinite loop if a message is larger than fetch size

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: core
    • Labels:
      None
    1. KAFKA-160_v2.patch
      4 kB
      Jun Rao
    2. KAFKA-160.patch
      2 kB
      Jun Rao

      Activity

      Hide
      Jun Rao added a comment -

      There were a couple of problems.
      1. We used to throw an exception while iterating messages in a ByteBufferMessageSet, if we can't iterate a single message out of the buffer. This indicates that either the fetch size is too small or if there is corruption in the data. Throwing exception is a good way to notify the consumer that something is wrong. After the compression patch, such an exception is no longer thrown.

      2. The constructor of ByetBufferMessageSet recently added a new parameter in the middle and some of the callers didn't get changed accordingly. So, some input parameters are misaligned.

      Attach a patch that fixes both problems.

      Show
      Jun Rao added a comment - There were a couple of problems. 1. We used to throw an exception while iterating messages in a ByteBufferMessageSet, if we can't iterate a single message out of the buffer. This indicates that either the fetch size is too small or if there is corruption in the data. Throwing exception is a good way to notify the consumer that something is wrong. After the compression patch, such an exception is no longer thrown. 2. The constructor of ByetBufferMessageSet recently added a new parameter in the middle and some of the callers didn't get changed accordingly. So, some input parameters are misaligned. Attach a patch that fixes both problems.
      Hide
      Jay Kreps added a comment -

      Hey did I break this?

      Show
      Jay Kreps added a comment - Hey did I break this?
      Hide
      Jun Rao added a comment -

      I think this was introduced when compression was added. I reviewed the patch, but didn't catch this.

      Show
      Jun Rao added a comment - I think this was introduced when compression was added. I reviewed the patch, but didn't catch this.
      Hide
      Jun Rao added a comment -

      Submitted patch v2, adding a unit test that uncovers the problem.

      Show
      Jun Rao added a comment - Submitted patch v2, adding a unit test that uncovers the problem.
      Hide
      Neha Narkhede added a comment -

      +1 on patch v2

      Show
      Neha Narkhede added a comment - +1 on patch v2
      Hide
      Jun Rao added a comment -

      Committed.

      Show
      Jun Rao added a comment - Committed.

        People

        • Assignee:
          Jun Rao
          Reporter:
          Jun Rao
        • Votes:
          0 Vote for this issue
          Watchers:
          0 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development