[KAFKA-10865] Improve trace-logging for Transformations (including Predicates)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: KafkaConnect
• Labels: newbie
• Assignee: Tiago Martins
• Reporter: Robin Moffatt
• Created: 18/Dec/20 14:37
• Updated: 05/Nov/21 20:08

Description

I've been spending a bunch of time poking around SMTs recently, and one common challenge I've had is being able to debug when things don't behave as I expect.
 
I know that there is the TransformationChain logger, but this only gives (IIUC) the input record

[2020-12-17 09:38:58,057] TRACE [sink-simulator-day12-00|task-0] Applying transformation io.confluent.connect.transforms.Filter$Value to SinkRecord{kafkaOffset=10551, timestampType=CreateTime} ConnectRecord{topic='day12-sys01', kafkaPartition=0, key=2c2ceb9b-8b31-4ade-a757-886ebfb7a398, keySchema=Schema{STRING}, value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01}, valueSchema=Schema{io.mdrogalis.Gen0:STRUCT}, timestamp=1608197938054, headers=ConnectHeaders(headers=)} (org.apache.kafka.connect.runtime.TransformationChain:47)

 
I think it would be really useful to also have trace level logging that included:

• the output of each transform
• the evaluation and result of any `predicate`s

I have been using com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector which is really useful for seeing the final record:

[2020-12-17 09:38:58,057] INFO [sink-simulator-day12-00|task-0] record.value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01} (com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkTask:50)

 
But doesn't include things like topic name (which is often changed by common SMTs)

Issue Links

• links to: GitHub Pull Request #11375