[SPARK-24768] Have a built-in AVRO data source implementation

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 10/Jul/18 17:21
• Updated: 09/Jun/19 23:35
• Resolved: 17/Sep/18 05:25

Description

Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines.  Using the external package https://github.com/databricks/spark-avro, Spark SQL can read and write the avro data. Making spark-Avro built-in can provide a better experience for first-time users of Spark SQL and structured streaming. We expect the built-in Avro data source can further improve the adoption of structured streaming. The proposal is to inline code from spark-avro package (https://github.com/databricks/spark-avro). The target release is Spark 2.4.  

Issue Links

• contains: SPARK-24741 Have a built-in AVRO data source implementation
• is duplicated by: SPARK-26062 Rename spark-avro external module to spark-sql-avro (to match spark-sql-kafka)
• links to: [Github] Pull Request #21742 (gengliangwang)
• links to: [Github] Pull Request #21801 (ueshin)
• links to: [Github] Pull Request #21841 (gengliangwang)
• links to: [Github] Pull Request #21866 (gengliangwang)

Sub-Tasks

• Add function `from_avro` and `to_avro`: Resolved
• Upgrade AVRO version from 1.7.7 to 1.8.2: Resolved
• support reading AVRO logical types - Date: Resolved
• support reading AVRO logical types - Timestamp with different precisions: Resolved
• support reading AVRO logical types - Decimal: Resolved
• support reading AVRO logical types - Duration: Resolved
• AVRO unit test: use SQLTestUtils and Replace deprecated methods: Resolved
• Add write benchmark for AVRO: Resolved
• Add API `.avro` in DataFrameReader/DataFrameWriter: Resolved
• Refactor Avro Serializer and Deserializer: Resolved
• Don't ignore files without .avro extension by default: Resolved
• Fix paths to resource files in AvroSuite: Resolved
• Support for parsing AVRO binary column: Resolved
• Supporting to convert a column into binary of AVRO format: Resolved
• New option - ignoreExtension: Resolved
• Gather all options into AvroOptions: Resolved
• Simplify schema serialization: Resolved
• New options - compression and compressionLevel: Resolved
• Remove implicit class AvroDataFrameWriter/AvroDataFrameReader: Resolved
• Use SerializableConfiguration in Spark util: Resolved
• Add mapping for built-in Avro data source: Resolved
• Use internal.Logging instead for logging: Resolved
• Avro: revise the output record namespace: Resolved
• Generate Avro Binary files in test suite: Resolved
• Validate user specified output schema: Resolved
• Make the mapping of com.databricks.spark.avro to built-in module configurable: Resolved
• Documentaion: AVRO data source guide: Resolved
• Remove sql configuration spark.sql.avro.outputTimestampType: Resolved
• Detect recursive reference in Avro schema and throw exception: Resolved
• Support parse mode option for function `from_avro`: Resolved
• Override method `prettyName` in `from_avro`/`to_avro`: Resolved
• Add read benchmark for Avro: Resolved
• Avro: Validate input and output schema: Resolved
• Allow user-specified output schema in function `to_avro`: Resolved
• Show Avro related API in documentation: Resolved