[STORM-2914] Remove enable.auto.commit support from storm-kafka-client

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.2.0
• Fix Version/s: 2.0.0, 1.2.0
• Component/s: storm-kafka-client
• Labels: pull-request-available
• Epic Link: Release Apache Storm 1.2.0
• Assignee: Stig Rohde Døssing
• Reporter: Stig Rohde Døssing
• Created: 27/Jan/18 14:03
• Updated: 06/Feb/18 04:51
• Resolved: 06/Feb/18 04:51

Description

The enable.auto.commit option causes the KafkaConsumer to periodically commit the latest offsets it has returned from poll(). It is convenient for use cases where messages are polled from Kafka and processed synchronously, in a loop.

Due to https://issues.apache.org/jira/browse/STORM-2913 we'd really like to store some metadata in Kafka when the spout commits. This is not possible with enable.auto.commit. I took at look at what that setting actually does, and it just causes the KafkaConsumer to call commitAsync during poll (and during a few other operations, e.g. close and assign) with some interval.

Ideally I'd like to get rid of ProcessingGuarantee.NONE, since I think ProcessingGuarantee.AT_MOST_ONCE covers the same use cases, and is likely almost as fast. The primary difference between them is that AT_MOST_ONCE commits synchronously.

If we really want to keep ProcessingGuarantee.NONE, I think we should make our ProcessingGuarantee.NONE setting cause the spout to call commitAsync after poll, and never use the enable.auto.commit option. This allows us to include metadata in the commit.

Issue Links

• is related to: STORM-2913 STORM-2844 made autocommit and at-most-once storm-kafka-client spouts log warnings on every emit, because those modes don't commit the right metadata to Kafka
• links to: GitHub Pull Request #2537