[STORM-1136] Provide a bin script to check consumer lag from KafkaSpout to Kafka topic offsets and integrate with storm ui

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0, 1.1.0
• Component/s: storm-kafka, storm-kafka-client, storm-kafka-monitor
• Labels: pull-request-available
• Epic Link: Release Apache Storm 1.1.0
• Assignee: Priyank Shah
• Reporter: Sriharsha Chintalapani
• Created: 27/Oct/15 20:08
• Updated: 14/Nov/17 17:29
• Resolved: 07/Jul/16 04:33

Description

We store kafkaspout offsets in zkroot + id path in zookeeper. Kafka provides a utility and a protocol request to fetch latest offsets into topic

example:
bin/kafka-run-classh.sh kafka.tools.GetOffsetTool 

we should provide a way for the user to check how far the kafka spout read into topic and whats the lag. If we can expose this via UI even better.

Issue Links

• is cloned by: STORM-1891 Incorporate trident kafka spout for the kafka offset lag tool
• links to: GitHub Pull Request #1451
• links to: GitHub Pull Request #1505