[KAFKA-7669] Stream topology definition is not robust to the ordering changes

• Type: Wish
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Mateusz Owczarek
• Created: 22/Nov/18 16:36
• Updated: 08/Feb/20 03:53

Description

It seems that if the user does not guarantee the order of the stream topology definition, he may end up with multiple stream branches having the same internal changelog (and repartition, if created) topic.

Let's assume:

val initialStream = new StreamsBuilder().stream(sth);
val someStrings = (1 to 10).map(_.toString)
val notGuaranteedOrderOfStreams: Map[String, KStream[...]] = someStrings.map(s => s -> initialStream.filter(...)).toMap

When the user defines now common aggregation logic for the notGuaranteedOrderOfStreams, and runs multiple instances of the application the KSTREAM-AGGREGATE-STATE-STORE topics names will not be unique and will contain results of the different streams from notGuaranteedOrderOfStreams map.

All of this without a single warning that the topology (or just the order of the topology definition) differs in different instances of the Kafka Streams application.

Also, I am concerned that ids in "KSTREAM-AGGREGATE-STATE-STORE-id-changelog " match so well for the different application instances (and different topologies).

 

Issue Links

• is related to: KAFKA-9518 NullPointerException on out-of-order topologies
• is related to: KAFKA-8307 Kafka Streams should provide some mechanism to determine topology equality and compatibility
• is related to: KAFKA-8810 Add mechanism to detect topology mismatch between streams instances