[KAFKA-6391] output from ensure copartitioning is not used for Cluster metadata, resulting in partitions without tasks working on them

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: streams
• Assignee: Clemens Valiente
• Reporter: Clemens Valiente
• Created: 20/Dec/17 15:39
• Updated: 02/Jan/18 18:22
• Resolved: 02/Jan/18 18:22

Description

https://github.com/apache/kafka/blob/1.0/streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamPartitionAssignor.java#L394

Map<TopicPartition, PartitionInfo> allRepartitionTopicPartitions is created from repartitionTopicMetadata
THEN we do ensureCoPartitioning on repartitionTopicMetadata
THEN we create topics and partitions according to repartitionTopicMetadata
THEN we use allRepartitionTopicPartitions to create our Cluster fullMetadata
THEN we use fullMetadata to assign the tasks and no longer use repartitionTopicMetadata

This results in any change to repartitionTopicMetadata in ensureCoPartitioning to be used for creating partitions but no tasks are ever created for any partition added by ensureCoPartitioning()

the fix is easy: First ensureCoPartitioning() on repartitionTopicMetadata before creating allRepartitionTopicPartitions

Issue Links

• links to: GitHub Pull Request #4347
• links to: GitHub Pull Request #4347