[HUDI-2320] Add support ByteArrayDeserializer in AvroKafkaSource

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: deltastreamer
• Labels: pull-request-available
• Assignee: 董可伦
• Reporter: 董可伦
• Created: 19/Aug/21 09:31
• Updated: 03/Nov/21 12:12
• Resolved: 03/Nov/21 12:12

Description

When the 'value.serializer' of Kafka Avro Producer is 'org.apache.kafka.common.serialization.ByteArraySerializer',Use the following configuration

--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
--schemaprovider-class org.apache.hudi.utilities.schema.JdbcbasedSchemaProvider \
--hoodie-conf "hoodie.deltastreamer.source.kafka.value.deserializer.class=org.apache.kafka.common.serialization.ByteArrayDeserializer"

For now,It will throw an exception::

java.lang.ClassCastException: [B cannot be cast to org.apache.avro.generic.GenericRecord

After support ByteArrayDeserializer,Use the configuration above，It works properly.And there is no need to provide 'schema.registry.url',For example, we can use the JdbcbasedSchemaProvider to get the sourceSchema

Issue Links

• links to: GitHub Pull Request #3502