[HIVE-21218] KafkaSerDe doesn't support topics created via Confluent Avro serializer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Component/s: kafka integration, Serializers/Deserializers
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Release Note: KafkaSerDe now supports Confluent Avro message format.
• Assignee: David McGinnis
• Reporter: Milan Baran
• Created: 05/Feb/19 15:45
• Updated: 18/Jun/20 04:48
• Resolved: 12/Mar/20 19:42

Description

According to Google groups the Confluent avro serialzier uses propertiary format for kafka value - <magic_byte 0x00><4 bytes of schema ID><regular avro bytes for object that conforms to schema>. 

This format does not cause any problem for Confluent kafka deserializer which respect the format however for hive kafka handler its bit a problem to correctly deserialize kafka value, because Hive uses custom deserializer from bytes to objects and ignores kafka consumer ser/deser classes provided via table property.

It would be nice to support Confluent format with magic byte.

Also it would be great to support Schema registry as well.

Issue Links

• links to: GitHub Pull Request #526
• links to: GitHub Pull Request #933