Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-10865

Improve trace-logging for Transformations (including Predicates)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • KafkaConnect

    Description

      I've been spending a bunch of time poking around SMTs recently, and one common challenge I've had is being able to debug when things don't behave as I expect.
       
      I know that there is the TransformationChain logger, but this only gives (IIUC) the input record

      [2020-12-17 09:38:58,057] TRACE [sink-simulator-day12-00|task-0] Applying transformation io.confluent.connect.transforms.Filter$Value to SinkRecord{kafkaOffset=10551, timestampType=CreateTime} ConnectRecord{topic='day12-sys01', kafkaPartition=0, key=2c2ceb9b-8b31-4ade-a757-886ebfb7a398, keySchema=Schema{STRING}, value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01}, valueSchema=Schema{io.mdrogalis.Gen0:STRUCT}, timestamp=1608197938054, headers=ConnectHeaders(headers=)} (org.apache.kafka.connect.runtime.TransformationChain:47)
      

       
      I think it would be really useful to also have trace level logging that included:

      • the output of each transform
      • the evaluation and result of any `predicate`s

      I have been using com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkConnector which is really useful for seeing the final record:

      [2020-12-17 09:38:58,057] INFO [sink-simulator-day12-00|task-0] record.value=Struct{units=16,product=Founders Breakfast Stout,amount=30.41,txn_date=Sat Dec 12 18:21:18 GMT 2020,source=SYS01} (com.github.jcustenborder.kafka.connect.simulator.SimulatorSinkTask:50)
      

       
      But doesn't include things like topic name (which is often changed by common SMTs)

      Attachments

        Issue Links

          Activity

            People

              upsidedownsmile Tiago Martins
              rmoff Robin Moffatt
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: