[KAFKA-346] Don't call commitOffsets() during rebalance

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7, 0.7.1, 0.8.0
• Component/s: core
• Assignee: Unassigned
• Reporter: Peter Romianowski
• Created: 18/May/12 14:57
• Updated: 08/Apr/15 01:23
• Resolved: 08/Apr/15 01:23

Description

A sample use-case

If I read the source correctly then offsets can be committed at any
time (whenever there is a change in consumer or broker zk registry).
Our application doesn't use auto-commit in order to batch some
messages together, process them and if everything went fine, we call
commitOffsets(). If, for any reason, the processing of messages does
not succeed, we rely on Kafka's promise to re-deliver the messages.

But if ZKRebalancerListener triggers a rebalance before our "batch" of
messages is full, then offsets will be committed even if the messages
have not been processed yet by our application. So if then processing
of these messages fails, we basically lost them, right?

(See discussion at http://www.mail-archive.com/[e-mail]/msg01415.html)

Issue Links

• depends upon: KAFKA-345 Add a listener to ZookeeperConsumerConnector to get notified on rebalance events