[KAFKA-9197] Consider introducing numberOfPartitions field to Grouped configuration class

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: streams
• Labels: needs-kip
• Assignee: Unassigned
• Reporter: Levani Kokhreidze
• Created: 16/Nov/19 11:58
• Updated: 16/Nov/19 20:55

Description

In the KIP-221 there was an idea of introducing number of partitions field to Grouped config class. During the discussion in the mailing list, couple of valid concerns were raised against this approach. 

Main argument against it was that, whenever user specifies number of partitions for internal, repartition topics, he/she really cares that those configurations will be applied. Case with group by is that, repartitioning will not happen, if key changing operation isn't performed. Therefore, number of partitions configuration specified by the user will never be applied. Alternatively, if user cares about manual repartitioning, one may do following in order to scale up/down sub topologies:

 

builder
  .stream("topic")
  .repartition((key, value) -> value.newKey(), Repartitioned.withNumberOfPartitions(5))       
  .groupByKey()       
  .count();

 

On the other hand, there were other valid arguments for adding numberOfPartitions field to Grouped config class. It was raised in the mailing list that, we should treat `numberOfPartitions` field as "desired" number of partitions specified by the user, so that if repartitioning is required, Kafka Streams must use value specified in there.

 

Idea of this ticket is to follow-up on this discussion and implement this feature if there's an actual need from the Kafka Streams users.

 

Issue Links

• mentioned in: Page Loading...