[STORM-2600] Improve or replace storm-kafka-monitor

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.0.0
• Component/s: storm-kafka-monitor
• Assignee: Unassigned
• Reporter: Stig Rohde Døssing
• Created: 25/Jun/17 15:26
• Updated: 30/Mar/19 13:09
• Resolved: 30/Mar/19 13:09

Description

The storm-kafka-monitor module, which is used by Storm UI to show offset lag for topologies with Kafka spouts, has some shortcomings:

• The Storm UI integration code doesn't seem to be able to support topic subscriptions that change after topology submission. The UI code (https://github.com/apache/storm/blob/64e29f365c9b5d3e15b33f33ab64e200345333e4/storm-core/src/jvm/org/apache/storm/utils/TopologySpoutLag.java#L91) gets the topic list it should request offset lag for via the spout's getComponentConfiguration method, as far as I can tell through this call https://github.com/apache/storm/blob/9e31509d47c4e91c1009f55c7ccf321d7d7e63aa/storm-client/src/jvm/org/apache/storm/topology/TopologyBuilder.java#L541. It seems like the component configuration is intended to be static once the topology has started running. This prevents us from showing the right topic list for subscriptions that are not known at submission time, which is currently the case for Pattern subscriptions. The topic list for that type of subscription isn't known until the spout has started the KafkaConsumer in ISpout.open(). I don't see a way to fix this, unless there is some way to update the component configuration when the subscription changes.
• The jar is installed along with the cluster, and depends on the Kafka version specified in Storm's root POM. Kafka guarantees backwards compatible client-server communication for one release only, so there's a potential coupling between Storm cluster version and Kafka version. If users want to update the Kafka version in storm-kafka-monitor, they have to rebuild that module and replace the jar in their Storm install.
• The UI integration uses the storm-kafka-monitor Bash script to start the monitoring code, in order to avoid a dependency between storm-core and storm-kafka-monitor. This prevents the UI integration from working on Windows. We could supply a Windows script as well, but then we'd need to keep the two in sync.

I am wondering if these problems could be solved by implementing offset lag monitoring via the metrics system instead. The spout could periodically seek to the log end offset and submit a metric for how far behind the committed offset is, then seek back to where it left off.

Issue Links

• Is contained by: STORM-3202 Include offset information to spout metrics and remove storm-kafka-monitor