Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8540

Improve InsertField connect transform to allow inserting processed timestamp

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.1
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:
    • Flags:
      Important

      Description

      We have requirement to insert a new field that gets populated with the timestamp when kafka message was processed by the connector.

      There is no such transform currently available. "InputField" transform provides timestamp.field option but that sets the field value to the timestamp when messaged landed in kafka and not when message was processed by the connector.

      Here is the proposal

      Enhance InputField transform to add a new option called "processedTimestamp.field" that will allow users to insert new field with value of message processed timestamp as shown in below configuration

      "transforms": "InsertField",
      "transforms.InsertField.type":"org.apache.kafka.connect.transforms.InsertField$Value",
      "transforms.InsertField.processedTimestamp.field": "field_name"

      here is the potential code block to set this value

      updatedValue.put(processedTimestampField.name, new Date()); 

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              khushal.kumthekar@gmail.com Khushal Kumthekar
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: