[KAFKA-18053] Clean up TopologyConfig and API for supplying configs needed by the topology

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams
• Labels: needs-kip
• Assignee: Unassigned
• Reporter: A. Sophie Blee-Goldman
• Created: 21/Nov/24 04:49
• Updated: 02/Dec/24 12:23

Description

As mentioned briefly in KIP-1112, the current API around configs which need to be applied prior to building the topology can be cumbersome and easy to implement incorrectly. This leads to awkward code and confused users at best, and applications that silently drop some configs at worst.

There are three main problems we want to solve:

1. The most serious issue is that the KafkaStreams constructor requires the application configs whereas the topology can be built without them, so most people will only pass their configs into KafkaStreams. However certain configs have to be passed into the topology as well, which is easy for users to miss and leads to a silent misconfiguration that can easily go unnoticed
2. Over time we've added several new APIs for configs that need to be applied at different/earlier stages of the topology building process. So now users are faced with multiple ways of passing these configs into the topology, and configs that may work with any of them or may need to be used with a specific API
3. At some point we also made the TopologyConfig class public, which was originally intended only for internal use. Furthermore it's only purpose was to separate configs between different topologies, a feature that has now been deprecated and will soon be removed, at which point TopologyConfig serves no purpose and should be removed as well. Unfortunately, the newest APIs for passing configs into a topology are a Topology and StreamsBuilder constructor that have a TopologyConfig  parameter, rather than the StreamsConfig parameter used in the previous API (ie StreamsBuilder#build). This is unnecessary since StreamsConfig would work just as well, and in fact a TopologyConfig has to be constructed from a StreamsConfig so users are just adding a wrapper layer

As of filing this ticket, the list of topology-specific configs is:

• topology.optimization
• processor.wrapper.class

However there are new configs in this category being proposed in other KIPs currently being discussed

Sub-Tasks

• Automatically detect missed configs needed by a topology: Open
• Remove TopologyConfig and clean up Topology/StreamsBuilder config APIs: Open