[STORM-631] Simplifying kafka connector code

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Component/s: storm-kafka
• Assignee: Hugo Da Cruz Louro
• Reporter: Parth Brahmbhatt
• Created: 20/Jan/15 06:37
• Updated: 02/Jul/24 09:53
• Resolved: 02/Jul/24 09:53

Description

The current kafka connector was written when kafka did not support any APIs to expose its internal structures. We already have STORM-590 to modify the kafka connector to new kafka meta APIS.

The current codebase uses some marker interfaces to support reading kafka partition/broker info from zookeeper or for users to specify the mapping them self. Adding one more layer that reads this info using kakfa APIs would reduce readability. Because we are using marker interfaces there are bunch of places that has if statements with instanceof checks and each one creates a separate code path to trace. We should delete these interfaces and their implementation in favor of a single way to get kafka topic information, using kafka APIs. This will be a backward incompatible change but should make future changes and usage easy.

Current kafka connector also has 2 separate configs for trident and core topologies. This configurations extend from a common interface but the extensions are not interchangeable. I think it is intuitive to have a single config with sane defaults for both core and trident spouts.