[KAFKA-5581] Avoid creating changelog topics for state stores that are materialized from a source topic

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: streams
• Labels: architecture performance
• Assignee: Jeyhun Karimov
• Reporter: Guozhang Wang
• Created: 11/Jul/17 02:07
• Updated: 15/May/20 02:14
• Resolved: 15/May/20 02:14

Description

Today Streams make all state stores to be backed by a changelog topic by default unless users overrides it by disableLogging when creating the state store / materializing the KTable. However there are a few cases where a separate changelog topic would not be required as we can re-use an existing topic for that. A few examples:

There are a few places where the materialized store do not need a separate changelog topic. This issue summarize a specific issue:

1) If a KTable is read directly from a source topic, and is materialized i.e.

table1 = builder.table("topic1", "store1")`.

In this case table1's changelog topic can just be topic1, and we do not need to create a separate table1-changelog topic.

2) if a KStream is materialized for joins where the streams are directly from a topic, e.g.:

stream1 = builder.stream("topic1");
stream2 = builder.stream("topic2");
stream3 = stream1.join(stream2, windows);  // stream1 and stream2 are materialized with a changelog topic

Issue Links

• is fixed by: KAFKA-6036 Enable logical materialization to physical materialization
• relates to: KAFKA-6761 Reduce Kafka Streams Footprint