Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.1
-
None
-
KafkaSerDe now supports Confluent Avro message format.
Description
According to Google groups the Confluent avro serialzier uses propertiary format for kafka value - <magic_byte 0x00><4 bytes of schema ID><regular avro bytes for object that conforms to schema>.
This format does not cause any problem for Confluent kafka deserializer which respect the format however for hive kafka handler its bit a problem to correctly deserialize kafka value, because Hive uses custom deserializer from bytes to objects and ignores kafka consumer ser/deser classes provided via table property.
It would be nice to support Confluent format with magic byte.
Also it would be great to support Schema registry as well.