[STORM-2270] Kafka spout should consume from latest when zk committed offset bigger than latest offset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0, 0.9.6
• Fix Version/s: 1.1.0
• Component/s: storm-kafka
• Labels: easyfix
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Unassigned
• Reporter: Danny Chen
• Created: 04/Jan/17 03:08
• Updated: 08/Feb/17 20:53
• Resolved: 08/Feb/17 20:53

Description

Kafka spout should consume from latest when ZK offset bigger than latest offset[ an TopicOffsetOutOfRangeException thrown out ], especially when Kafka topic change it's leader and some data lost, if we consume from earliest offset, much meaningless duplicate records will be re-consumed. So, we should consume from latest offset instead.

Issue Links

• links to: GitHub Pull Request #1851