[STORM-2691] storm-kafka-client Trident spout implements the Trident interface incorrectly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0, 1.1.1
• Fix Version/s: 2.0.0
• Component/s: storm-kafka-client
• Labels: pull-request-available
• Epic Link: Release Apache Storm 2.0.0
• Assignee: Stig Rohde Døssing
• Reporter: Stig Rohde Døssing
• Created: 08/Aug/17 22:54
• Updated: 19/Feb/19 17:06
• Resolved: 01/Mar/18 10:46

Description

The Trident Kafka spout uses the KafkaTridentSpoutTopicPartitionRegistry enum to pass existing topic partitions from the spout to the coordinator. This only works when those components happen to be in the same JVM, because the coordinator gets the topic information from the KafkaConsumer started by KafkaTridentSpoutEmitter.

The coordinator runs in the TridentSpoutCoordinator bolt here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L162, while the spout instances (emitters) run in TridentSpoutExecutors here https://github.com/apache/storm/blob/4c8a986f519cdf3e63bed47e9c4f723e4867267a/storm-client/src/jvm/org/apache/storm/trident/topology/TridentTopologyBuilder.java#L176.

We should replace the registry enum with writes to Zookeeper or something similar.

Edit: The fix for this is likely to be a broader change where we split the Subscription API into a few parts so the assignment process can be split across the coordinator and emitter instead of the emitter doing everything.

Issue Links

• contains: STORM-3337 KafkaTridentSpoutOpaque can lose offset data in Zookeeper
• is a clone of: STORM-2473 KafkaTridentSpoutOpaque's implementation is incorrect.
• is duplicated by: STORM-2719 Trident Kafka Spout Emitters do not get full partition information in getOrderedPartitions()
• links to: GitHub Pull Request #2300