[STORM-2869] KafkaSpout discards all pending records when adjusting the consumer position after a commit

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.2.0
• Fix Version/s: 2.0.0, 1.2.0
• Component/s: storm-kafka-client
• Labels: pull-request-available
• Epic Link: Release Apache Storm 1.2.0
• Assignee: Stig Rohde Døssing
• Reporter: Stig Rohde Døssing
• Created: 26/Dec/17 22:16
• Updated: 29/Dec/17 02:19
• Resolved: 29/Dec/17 02:19

Description

As part of the STORM-2666 fix the spout clears out waitingToEmit when the consumer position falls behind the committed offset during a commit. We only need to do it for the affected partition, and then only for the records that are behind the committed offset.

Also the validation check in emitOrRetryTuple is slightly too permissive, it should check whether the current record is behind the committed offset, not whether the consumer position is behind the committed offset.

Issue Links

• links to: GitHub Pull Request #2481
• links to: GitHub Pull Request #2484