[SPARK-24768] Have a built-in AVRO data source implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 10/Jul/18 17:21
• Updated: 09/Jun/19 23:35
• Resolved: 17/Sep/18 05:25

Description

Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines.  Using the external package https://github.com/databricks/spark-avro, Spark SQL can read and write the avro data. Making spark-Avro built-in can provide a better experience for first-time users of Spark SQL and structured streaming. We expect the built-in Avro data source can further improve the adoption of structured streaming. The proposal is to inline code from spark-avro package (https://github.com/databricks/spark-avro). The target release is Spark 2.4.  

Issue Links

• contains: SPARK-24741 Have a built-in AVRO data source implementation
• is duplicated by: SPARK-26062 Rename spark-avro external module to spark-sql-avro (to match spark-sql-kafka)
• links to: [Github] Pull Request #21742 (gengliangwang)
• links to: [Github] Pull Request #21801 (ueshin)
• links to: [Github] Pull Request #21841 (gengliangwang)
• links to: [Github] Pull Request #21866 (gengliangwang)