-
Type:
Bug
-
Status: Patch Available
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 3.1.1
-
Fix Version/s: None
-
Component/s: kafka integration, Serializers/Deserializers
-
Labels:
-
Target Version/s:
-
Release Note:KafkaSerDe now support Confluent message format.
According to Google groups the Confluent avro serialzier uses propertiary format for kafka value - <magic_byte 0x00><4 bytes of schema ID><regular avro bytes for object that conforms to schema>.
This format does not cause any problem for Confluent kafka deserializer which respect the format however for hive kafka handler its bit a problem to correctly deserialize kafka value, because Hive uses custom deserializer from bytes to objects and ignores kafka consumer ser/deser classes provided via table property.
It would be nice to support Confluent format with magic byte.
Also it would be great to support Schema registry as well.
- links to