Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2054 Fix Kafka transient unit test failures
  3. KAFKA-2342

KafkaConsumer rebalance with in-flight fetch can cause invalid position

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 0.9.0.0
    • Component/s: core
    • Labels:
      None

      Description

      If a rebalance occurs with an in-flight fetch, the new KafkaConsumer can end up updating the fetch position of a partition to an offset which is no longer valid. The consequence is that we may end up either returning to the user messages with an unexpected position or we may fail to give back the right offset in position().

      Additionally, this bug causes transient test failures in ConsumerBounceTest.testConsumptionWithBrokerFailures with the following exception:

      kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED
      java.lang.NullPointerException
      at org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:949)
      at kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:86)
      at kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:61)

        Attachments

          Activity

            People

            • Assignee:
              hachikuji Jason Gustafson
              Reporter:
              junrao Jun Rao
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: