[KAFKA-13913] Provide builders for KafkaProducer/KafkaConsumer and KafkaStreams

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.2.0
• Component/s: clients, streams
• Labels: kip
• Assignee: François Rosière
• Reporter: François Rosière
• Created: 18/May/22 20:03
• Updated: 14/Apr/23 06:47

Description

To have more flexibility, builders should be provided for the following objects

• KafkaProducer
• KafkaConsumer
• KafkaStreams 

These builders will give an easy way to construct these objects using different arguments/combinations without having to add a new constructor every time a new parameter is required.

They will also allow using already configured dependencies coming from an injection framework such as Spring (see https://github.com/spring-projects/spring-kafka/issues/2244).

From a user point of view, builders would be used as follow

KafkaProducer kafkaProducer = new KafkaProducerBuilder<String, MyPojo(<MAP_OR_PROPERTIES_OR_CONFIG>)  
  .withKeySerializer(<KEY_SERIALIZER>)  
  .withValueSerializer(<VALUE_SERIALIZER>)  
  .withInterceptors(<LIST_OF_INTERCEPTORS>)  
  .withPartitioner(<PARTITIONER>) 
  .withMetricsReporter(<METRICS_REPORTER>)  
  .build();  

KafkaConsumer consumer = new KafkaConsumerBuilder<String, MyPojo(<MAP_OR_PROPERTIES_OR_CONFIG>)  
  .withKeyDeserializer(<KEY_DESERIALIZER>)  
  .withValueDeserializer(<VALUE_DESERIALIZER>)  
  .withInterceptors(<LIST_OF_INTERCEPTORS>)  
  .withMetricsReporter(<METRICS_REPORTER>)  
  .build(); 

KafkaStreams kafkaStreams = new KafkaStreamsBuilder(<TOPOLOGY>, <MAP_OR_PROPERTIES_OR_CONFIG>)  
  .withProducerInterceptors(<LIST_OF_PRODUER_INTERCEPTORS>)  
  .withConsumerInterceptors(<LIST_OF_CONSUMER_INTERCEPTORS>)  
  .withTime(<TIME>)  .withKafkaClientSupplier(<KAFKA_CLIENT_SUPPLIER>)  
  .withMetricsReporter(<METRICS_REPORTER>)  
  .build();

This KIP can be seen as the continuity of the KIP-832.

More details can be found in the related KIP https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211884640

Issue Links

• contains: KAFKA-13864 Change the visibility of a KafkaProducer and KafkaConsumer constructor
• duplicates: KAFKA-3943 ConfigDef should support a builder pattern.
• duplicates: KAFKA-4436 Provide builder pattern for StreamsConfig
• is duplicated by: KAFKA-14373 provide builders for producer/consumer