[HUDI-837] Fix AvroKafkaSource to use the latest schema for reading

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: deltastreamer
• Labels: pull-request-available sev:critical user-support-issues
• Assignee: sivabalan narayanan
• Reporter: Pratyaksh Sharma
• Created: 24/Apr/20 21:07
• Updated: 06/Apr/21 01:55
• Resolved: 23/Mar/21 15:42

Description

Currently we specify KafkaAvroDeserializer as the value for value.deserializer in AvroKafkaSource. This implies the published record is read using the same schema with which it was written even though the schema got evolved in between. As a result, messages in incoming batch can have different schemas. This has to be handled at the time of actually writing records in parquet. 

This Jira aims at providing an option to read all the messages with the same schema by implementing a new custom deserializer class. 

Issue Links

• is depended upon by: HUDI-901 Bug Bash 0.6.0 Tracking Ticket
• links to: GitHub Pull Request #1562
• links to: GitHub Pull Request #1562