[KAFKA-13638] Slow KTable update when forwarding multiple values from transformer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 3.1.0, 3.0.0
• Component/s: streams
• Assignee: Unassigned
• Reporter: Ulrik
• Created: 02/Feb/22 12:53
• Updated: 10/Aug/22 15:18
• Resolved: 10/Aug/22 15:18

Description

I have a topology where I stream messages from an input topic, transform the message to multiple messages (via context.forward), and then store those messages in a KTable.

Since upgrading from kafka-streams 2.8.1 to 3.1.0 I have noticed that my tests take significantly longer time to run. 

 

I have attached a test class to demonstrate my scenario. When running this test with kafka-streams versions 2.8.1 and 3.1.0 I came up with the following numbers:

 

Version 2.8.1

• one input message and one output message: 541 ms
• 8 input message and 30 output message per input message (240 output messages in total): 919 ms

 

Version 3.1.0

• one input message and one output message: 908 ms
• 8 input message and 30 output message per input message (240 output messages in total): 6 sec 94 ms

 

Even when the transformer just transforms and forwards one input message to one output message, the test takes approx. 400 ms longer to run.

When transforming 8 input messages to 240 output messages it takes approx 5 seconds longer.