Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-9782

Kafka Connect InsertField transform - Add the ability to insert event's Key into Value

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
      None

      Description

      I'm using Debezium to pull change data capture events from a Mongo DB and write them to S3 with the Confluent S3 Sink. The problem is Debezium stores the document's key in each event's key and the S3 connector discards this key. I need the ability to insert the key as a new field in the event value. It seems that this would fit in perfectly into the InsertField transform or create a new transform KeyToValue.

      Here is an example of someone else running into this same limitation and creating a custom transform.
      https://gist.github.com/shashidesai/aaf72489165c6a0fd73a3b51e5a8892a

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Tomczik76 Ryan Tomczik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: