[KAFKA-4299] Consumer offsets reset for all topics after increasing partitions for one topic

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.10.0.1
• Assignee: Unassigned
• Reporter: Juho Autio
• Created: 13/Oct/16 08:32
• Updated: 17/Oct/16 10:26
• Resolved: 17/Oct/16 10:25

Description

I increased partitions for one existing topic (2->10), but was surprised to see that it entirely reset the committed offsets of my consumer group.

All topics & partitions were reset to the earliest offset available, and the consumer read everything again.

Documentation doesn't mention anything like this. Is this how it's supposed to work, or a bug?

I would've expected the consumer offsets to not decrease at all, especially for the topics that I didn't even touch.

For the altered topic I would've expected that consuming the previously existing partitions 0 and 1 would've continued from the position where they were, and naturally starting to read the new added partitions from 0.

I added partitions according to the "Modifying topics" section of Kafka 0.10.0 Documentation:

To add partitions you can do

 > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic altered_topic --partitions 10

Previously this topic had 2 partitions.

For the consumer I'm using kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter().

And version is:

<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>0.10.0.1</version>

Kafka cluster itself is kafka_2.11-0.10.0.1.

This is quite problematic because we can't afford waiting for consumers to read the full buffer from the beginning (for all topics!) when increasing partitions for a topic.

Some possibly relevant settings we have for the consumer:

kafka.partition.assignment.strategy = "range"
kafka.auto.offset.reset = "smallest"
kafka.auto.commit.enable = "false"
kafka.offsets.storage = "kafka"
kafka.dual.commit.enabled = false
kafka.consumer.timeout.ms = "2000"
kafka.auto.create.topics.enable = true