[SPARK-25789] Support for Dataset of Avro

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Aleksander Eskilson
• Shepherd: Michael Armbrust
• Created: 20/Oct/18 16:29
• Updated: 13/Feb/24 23:18

Description

Support for Dataset of Avro records in an API that would allow the user to provide a class to an Encoder for Avro, analogous to the Bean encoder. This functionality was previously to be provided by SPARK-22739 and Spark-Avro #169. Avro functionality was folded into Spark-proper by SPARK-24768, eliminating the need to maintain a separate library for Avro in Spark. Resolution of this issue would:

• Add necessary Expression elements to Spark
• Add an AvroEncoder for Datasets of Avro records to Spark

Issue Links

• links to: [Github] Pull Request #22878 (xuanyuanking)
• links to: GitHub Pull Request #22878