Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4380

Add Kafka Key attribute to outgoing FF for ConsumeKafkaRecord

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Often times during processing of data having access to the Key used for the message in Kafka is important. For example, some CDC tools that use Kafka as the destination for changes will use the primary key for the table as the key when writing data to Kafka. The value of this key becomes import after the is consumed from Kakfa, as many times the final destination is another data store, for which you need to know which column(s) of data represents the key.

      When you introduce a Schema Registry into the picture then many times the key is written into Kafka is often encoded, for example, if Avro is being used for data serialization into Kafka then the key may have its own schema that is serialized with. Thus, this change would need to ensure that the key is deserialized before being written as an attribute.

      Not sure if it gets overly complex or if it is required to have Reader/Writer configuration for the key as well as the value. It is certainly to be expected that is the key and value are stored in Kafka in Avro that the schema for each will be unique.

        Activity

        Hide
        sundara m sundaramoorthy M added a comment -

        Team,

        I would recommend to have the schema ID and VERSION Number also sent as a part of the flow file attribute when consuming data from Kafka.

        This will aid to help the schema evolution strategy OR to identify the schema change.

        Hope this can be accommodated in a single jira.

        Thanks,
        Sundar.

        Show
        sundara m sundaramoorthy M added a comment - Team, I would recommend to have the schema ID and VERSION Number also sent as a part of the flow file attribute when consuming data from Kafka. This will aid to help the schema evolution strategy OR to identify the schema change. Hope this can be accommodated in a single jira. Thanks, Sundar.

          People

          • Assignee:
            Unassigned
            Reporter:
            apsaltis Andrew Psaltis
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development