Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
Often times during processing of data having access to the Key used for the message in Kafka is important. For example, some CDC tools that use Kafka as the destination for changes will use the primary key for the table as the key when writing data to Kafka. The value of this key becomes import after the is consumed from Kakfa, as many times the final destination is another data store, for which you need to know which column(s) of data represents the key.
When you introduce a Schema Registry into the picture then many times the key is written into Kafka is often encoded, for example, if Avro is being used for data serialization into Kafka then the key may have its own schema that is serialized with. Thus, this change would need to ensure that the key is deserialized before being written as an attribute.
Not sure if it gets overly complex or if it is required to have Reader/Writer configuration for the key as well as the value. It is certainly to be expected that is the key and value are stored in Kafka in Avro that the schema for each will be unique.