[HOP-3687] Allow reading of non-string messages with a Kafka Consumer

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Migrated to GHI
• Component/s: API, Beam, Transforms
• Assignee: Unassigned
• Reporter: Matt Casters
• Created: 18/Jan/22 15:36
• Updated: 20/Mar/23 17:37
• Resolved: 20/Mar/23 17:37

Description

For both the regular Kafka Consumer and the Beam Kafka Consumer transforms it would be great to be able to read non-String messages, for example in Avro format.

For the Beam Kafka Consumer I think it would be cool if you could specify the output format as "Avro Record" (in our example) or Serializable, Binary, with String as a default. 

This would allow us to set Kafka consumer option value.deserializer to KafkaAvroDeserializer and get an actual Avro Record as a message.