Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-364 Consumer re-design
  3. KAFKA-360

Add ability to disable rebalancing in ZooKeeper consumer

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      There is a need for a ZooKeeper-based Kafka consumer that does not re-balance. This is needed because we may be handling partitioning outside of Kafka. For example, I may have a stateful process that is meant to consume only from Partition 7 of a given Kafka topic. When that process stops, I don't want another consumer to take over partition 7.

      The benefits we get from using the ZooKeeper-based consumer (vs the Simple Consumer) without rebalancing is that offsets will still be handled by Kafka/ZK, as will failover when a partition's leader disappears/fails.

      I think the way to do this is to add a consumer config parameter that disables a consumer group's rebalancing. That way, the first consumer in the group to connect (when the ZK node is created) can specify if rebalancing should be enabled for the consumer group. If rebalancing is disabled, the consumers should be forced to supply a list of partition IDs that they wish to read from. Perhaps this can be done during the createMessageStreams call?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                criccomini Chris Riccomini
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: