Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-2361

Kafka spout - after topic leader change, it stops committing offsets to ZK

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 2.0.0, 1.1.0, 1.0.4
    • Component/s: storm-kafka
    • Labels:
      None

      Description

      After STORM-2296 although Kafka spouts do not generate duplicates, the offsets committment to ZK may stop on recreated PartitionManagers.

      This is because ack's for messages emitted by already destroyed PartitionManagers are not routed properly to the new PartitionManagers handling that partition.

      E.g:

       public void ack(Object msgId) {
              KafkaMessageId id = (KafkaMessageId) msgId;
              PartitionManager m = _coordinator.getManager(id.partition);
              if (m != null) {
                  m.ack(id.offset);
              }
      

      id.partition is Partition(host, partition, topic), which is different if Kafka broker changed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ernisv Ernestas Vaiciukevičius
                Reporter:
                ernisv Ernestas Vaiciukevičius
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m