[KAFKA-2342] KafkaConsumer rebalance with in-flight fetch can cause invalid position

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0.0
• Fix Version/s: 0.9.0.0
• Component/s: core
• Assignee: Jason Gustafson
• Reporter: Jun Rao
• Created: 17/Jul/15 00:11
• Updated: 22/Jul/15 20:10
• Resolved: 22/Jul/15 20:00

Description

If a rebalance occurs with an in-flight fetch, the new KafkaConsumer can end up updating the fetch position of a partition to an offset which is no longer valid. The consequence is that we may end up either returning to the user messages with an unexpected position or we may fail to give back the right offset in position().

Additionally, this bug causes transient test failures in ConsumerBounceTest.testConsumptionWithBrokerFailures with the following exception:

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures FAILED
java.lang.NullPointerException
at org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:949)
at kafka.api.ConsumerBounceTest.consumeWithBrokerFailures(ConsumerBounceTest.scala:86)
at kafka.api.ConsumerBounceTest.testConsumptionWithBrokerFailures(ConsumerBounceTest.scala:61)