[KAFKA-1787] Consumer should delete offsets and release partition ownership for deleted topics

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Won't Fix
• Affects Version/s: 0.8.2.0
• Fix Version/s: 0.8.2.0
• Assignee: Onur Karaman
• Reporter: Joel Jacob Koshy
• Created: 19/Nov/14 21:40
• Updated: 02/Dec/14 07:06
• Resolved: 02/Dec/14 07:06

Description

Marking as a blocker for now, but we can evaluate.

If a topic is deleted, the consumer currently does not rebalance (handleDataDeleted is a TODO in ZooKeeperConsumerConnector)

As a result, it will continue owning that (deleted) partition and continue to issue fetch requests. Those fetch requests will result in an UnknownTopicOrPartition error and thus will result in TopicMetadataRequests issued by the leader finder thread which can recreate the topic if auto-create is turned on.

Furthermore if we don't delete the offsets it is possible to lose messages if the topic is recreated immediately and the previously checkpointed offset (for the old data) is small. E.g., if a consumer is at offset 10 on some partition, and if the partition is deleted and recreated and 15 messages are sent to it then the first 10 messages will be lost.