[STORM-2340] fix AutoCommitMode issue in KafkaSpout

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0, 1.1.0
• Component/s: storm-kafka-client
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Mingmin Xu
• Reporter: Mingmin Xu
• Created: 03/Feb/17 18:42
• Updated: 14/Feb/17 06:29
• Resolved: 14/Feb/17 02:58

Description

What's the issue?
When KafkaSpout is running with AutoCommitMode=true, and ack executor=0, KafkaSpout cannot emit records after some time.

What's the cause?
Function poll() controls whether it need to pull more data from Kafka cluster. With the condition !waitingToEmit() && numUncommittedOffsets < maxUncommittedOffsets, it's always FALSE after reaching threshold, as numUncommittedOffsets is creasing and never reset on 'At-Most-Once' mode.

What's the solution?
I change the condition to !waitingToEmit() && (numUncommittedOffsets < maxUncommittedOffsets || consumerAutoCommitMode), that return TRUE with AutoCommitMode regarding of UnCommittedOffset, see the comments inline.
Also, it's not required to track emitted(msgId) and numUncommittedOffsets with AutoCommitMode .

Issue Links

• links to: GitHub Pull Request #1863
• links to: GitHub Pull Request #1919