Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-4380

Add Kafka Key attribute to outgoing FF for ConsumeKafkaRecord


    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      Often times during processing of data having access to the Key used for the message in Kafka is important. For example, some CDC tools that use Kafka as the destination for changes will use the primary key for the table as the key when writing data to Kafka. The value of this key becomes import after the is consumed from Kakfa, as many times the final destination is another data store, for which you need to know which column(s) of data represents the key.

      When you introduce a Schema Registry into the picture then many times the key is written into Kafka is often encoded, for example, if Avro is being used for data serialization into Kafka then the key may have its own schema that is serialized with. Thus, this change would need to ensure that the key is deserialized before being written as an attribute.

      Not sure if it gets overly complex or if it is required to have Reader/Writer configuration for the key as well as the value. It is certainly to be expected that is the key and value are stored in Kafka in Avro that the schema for each will be unique.




            • Assignee:
              apsaltis Andrew Psaltis
              apsaltis Andrew Psaltis
            • Votes:
              1 Vote for this issue
              2 Start watching this issue


              • Created: