Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-7144

Kafka Streams doesn't properly balance partition assignment

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 2.0.1, 2.1.0
    • Component/s: streams
    • Labels:
      None

      Description

      Kafka Streams doesn't always spread the tasks across all available instances/threads

      I have a topology which consumes a single partition topic and goes .through() a 12 partition topic. The makes 13 partitions.

       

      I then started 2 instances of the application. I would have expected the 13 partitions to be split across the 2 instances roughly evenly (7 partitions on one, 6 partitions on the other).

      Instead, one instance gets 12 partitions, and the other instance gets 1 partition.

       

      Repro case attached. I ran it a couple times, and it was fairly repeatable.

      Setup for the repro:

      $ ./bin/kafka-topics.sh --zookeeper localhost --create --topic one --partitions 1 --replication-factor 1 
      $ ./bin/kafka-topics.sh --zookeeper localhost --create --topic twelve --partitions 12 --replication-factor 1
      $ echo foo | kafkacat -P -b 127.0.0.1 -t one
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                bbejeck Bill Bejeck
                Reporter:
                wushujames James Cheng
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: