[KAFKA-12957] Refactor Streams Logical Plan Generation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Assignee: Unassigned
• Reporter: Guozhang Wang
• Created: 16/Jun/21 16:31
• Updated: 02/Mar/23 15:49

Description

There is a general issue of Streams logical plan -> physical plan generation, where the physical processor nodes are generated at the parsing phase rather than the logical plan compilation phase. The former stage is agnostic to any user configurations while only the latter stage have access to it, and hence we would not generate physical processor nodes during the parsing phase (i.e. any code related to StreamsBuilder), but defer them to the logical plan phase (i.e. XXNode.writeToTopology). This has several issues such that many physical processor instantiation requires to access the configs, and hence we have to defer it to the `init` procedure of the node, which is scattered in many places from logical nodes to physical processors.

This would be a big refactoring on Stream's logical plan generation, but I think it would worth to get this in a cleaner state.

Issue Links

• is blocked by: KAFKA-13336 Migrate StreamsBuilder/Topology class to interfaces and move Topology parameter from KafkaStreams constructor to #start