[HIVE-21218] KafkaSerDe doesn't support topics created via Confluent Avro serializer

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.1
• Component/s: kafka integration, Serializers/Deserializers
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Release Note: KafkaSerDe now support Confluent message format.
• Assignee: Milan Baran
• Reporter: Milan Baran
• Created: 05/Feb/19 15:45
• Updated: 15/Apr/19 14:57

Description

According to Google groups the Confluent avro serialzier uses propertiary format for kafka value - <magic_byte 0x00><4 bytes of schema ID><regular avro bytes for object that conforms to schema>. 

This format does not cause any problem for Confluent kafka deserializer which respect the format however for hive kafka handler its bit a problem to correctly deserialize kafka value, because Hive uses custom deserializer from bytes to objects and ignores kafka consumer ser/deser classes provided via table property.

It would be nice to support Confluent format with magic byte.

Also it would be great to support Schema registry as well.

Issue Links

• links to: GitHub Pull Request #526