[KAFKA-7144] Kafka Streams doesn't properly balance partition assignment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: streams
• Assignee: Bill Bejeck
• Reporter: James Cheng
• Created: 10/Jul/18 05:12
• Updated: 01/Aug/18 16:49
• Resolved: 25/Jul/18 05:00

Description

Kafka Streams doesn't always spread the tasks across all available instances/threads

I have a topology which consumes a single partition topic and goes .through() a 12 partition topic. The makes 13 partitions.

 

I then started 2 instances of the application. I would have expected the 13 partitions to be split across the 2 instances roughly evenly (7 partitions on one, 6 partitions on the other).

Instead, one instance gets 12 partitions, and the other instance gets 1 partition.

 

Repro case attached. I ran it a couple times, and it was fairly repeatable.

Setup for the repro:

$ ./bin/kafka-topics.sh --zookeeper localhost --create --topic one --partitions 1 --replication-factor 1 
$ ./bin/kafka-topics.sh --zookeeper localhost --create --topic twelve --partitions 12 --replication-factor 1
$ echo foo | kafkacat -P -b 127.0.0.1 -t one

Issue Links

• links to: GitHub Pull Request #5390