Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-1787

Consumer should delete offsets and release partition ownership for deleted topics

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 0.8.2.0
    • Fix Version/s: 0.8.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Marking as a blocker for now, but we can evaluate.

      If a topic is deleted, the consumer currently does not rebalance (handleDataDeleted is a TODO in ZooKeeperConsumerConnector)

      As a result, it will continue owning that (deleted) partition and continue to issue fetch requests. Those fetch requests will result in an UnknownTopicOrPartition error and thus will result in TopicMetadataRequests issued by the leader finder thread which can recreate the topic if auto-create is turned on.

      Furthermore if we don't delete the offsets it is possible to lose messages if the topic is recreated immediately and the previously checkpointed offset (for the old data) is small. E.g., if a consumer is at offset 10 on some partition, and if the partition is deleted and recreated and 15 messages are sent to it then the first 10 messages will be lost.

        Attachments

          Activity

            People

            • Assignee:
              onurkaraman Onur Karaman
              Reporter:
              jjkoshy Joel Koshy
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: