Kafka
  1. Kafka
  2. KAFKA-111

A bug in the iterator of the ByteBufferMessageSet returns incorrect offsets when it encounters a compressed empty message set

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.7
    • Component/s: None
    • Labels:
      None

      Description

      The deep iterator logic in the ByteBufferMessageSet returns incorrect offsets when it encounters empty compressed data. Ideally, it should be able to decompress the data, figure out that it is somehow empty, skip it and proceed to decoding rest of the data. To make this possible, the manner in which we update the offset to be returned by the iterator, needs to be tweaked.

      1. KAFKA-111.patch
        3 kB
        Neha Narkhede
      2. kafka-111.patch.v2
        3 kB
        Jun Rao
      3. kafka-111.patch.v3
        3 kB
        Jun Rao

        Activity

        Hide
        Neha Narkhede added a comment -

        This patch corrects the offset management in the deep iterator of the ByteBufferMessageSet, to handle the case of compressed empty message sets.

        Show
        Neha Narkhede added a comment - This patch corrects the offset management in the deep iterator of the ByteBufferMessageSet, to handle the case of compressed empty message sets.
        Hide
        Jun Rao added a comment -

        1. Please remove unreferenced imports before checking in.
        2. The following unit test now hangs.
        Test Starting: testCompression(kafka.consumer.ZookeeperConsumerConnectorTest)

        Show
        Jun Rao added a comment - 1. Please remove unreferenced imports before checking in. 2. The following unit test now hangs. Test Starting: testCompression(kafka.consumer.ZookeeperConsumerConnectorTest)
        Hide
        Jun Rao added a comment -

        The patch in kafka-108 also identifies this problem. The problem is that we need to set the new offset in 2 cases: (1) if the compressed unit is empty; (2) otherwise, the last message of the compressed unit is being iterated. This patches covers (1), but not (2). One solution is probably to use the code that we had before, but treat case (1) specially, i.e., we won't even create an inner iterator and just advance the offset.

        Show
        Jun Rao added a comment - The patch in kafka-108 also identifies this problem. The problem is that we need to set the new offset in 2 cases: (1) if the compressed unit is empty; (2) otherwise, the last message of the compressed unit is being iterated. This patches covers (1), but not (2). One solution is probably to use the code that we had before, but treat case (1) specially, i.e., we won't even create an inner iterator and just advance the offset.
        Hide
        Jun Rao added a comment -

        How about patch v2? It currently breaks PrimitiveApiTest. However, the test probably should be fixed since it includes an empty message set.

        Show
        Jun Rao added a comment - How about patch v2? It currently breaks PrimitiveApiTest. However, the test probably should be fixed since it includes an empty message set.
        Hide
        Neha Narkhede added a comment -

        Yes, looks like patch v2 is a better way of handling the above 2 cases. We should check what breaks with PrimitiveApiTest though, maybe just a test bug.

        Show
        Neha Narkhede added a comment - Yes, looks like patch v2 is a better way of handling the above 2 cases. We should check what breaks with PrimitiveApiTest though, maybe just a test bug.
        Hide
        Jun Rao added a comment -

        Patch v3. Fix test in kafka.javaapi.message.ByteBufferMessageSetTest.

        PrimitiveApiTest for javaapi still fails. However, it should pass after kafka-109 is fixed.

        Show
        Jun Rao added a comment - Patch v3. Fix test in kafka.javaapi.message.ByteBufferMessageSetTest. PrimitiveApiTest for javaapi still fails. However, it should pass after kafka-109 is fixed.

          People

          • Assignee:
            Unassigned
            Reporter:
            Neha Narkhede
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development