[STORM-2361] Kafka spout - after topic leader change, it stops committing offsets to ZK

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 2.0.0, 1.1.0, 1.0.4
• Component/s: storm-kafka
• Assignee: Ernestas Vaiciukevičius
• Reporter: Ernestas Vaiciukevičius
• Created: 14/Feb/17 15:36
• Updated: 14/Dec/17 00:55
• Resolved: 09/Mar/17 17:01

Description

After STORM-2296 although Kafka spouts do not generate duplicates, the offsets committment to ZK may stop on recreated PartitionManagers.

This is because ack's for messages emitted by already destroyed PartitionManagers are not routed properly to the new PartitionManagers handling that partition.

E.g:

 public void ack(Object msgId) {
        KafkaMessageId id = (KafkaMessageId) msgId;
        PartitionManager m = _coordinator.getManager(id.partition);
        if (m != null) {
            m.ack(id.offset);
        }

id.partition is Partition(host, partition, topic), which is different if Kafka broker changed.

Issue Links

• relates to: STORM-2296 Kafka spout - no duplicates on topic leader changes
• links to: GitHub Pull Request #1940