Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24768

Have a built-in AVRO data source implementation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

      Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines.  Using the external package https://github.com/databricks/spark-avro, Spark SQL can read and write the avro data. Making spark-Avro built-in can provide a better experience for first-time users of Spark SQL and structured streaming. We expect the built-in Avro data source can further improve the adoption of structured streaming. The proposal is to inline code from spark-avro package (https://github.com/databricks/spark-avro). The target release is Spark 2.4.  

      Attachments

        Issue Links

          contains

          New Feature - A new feature of the product, which has yet to be developed. SPARK-24741 Have a built-in AVRO data source implementation

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-26062 Rename spark-avro external module to spark-sql-avro (to match spark-sql-kafka)

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Pull request #21742 [Github] Pull Request #21742 (gengliangwang)

          Pull request #21801 [Github] Pull Request #21801 (ueshin)

          Pull request #21841 [Github] Pull Request #21841 (gengliangwang)

          Pull request #21866 [Github] Pull Request #21866 (gengliangwang)

          1.
          		 Add function `from_avro` and `to_avro` Sub-task Resolved Gengliang Wang
          2.
          		 Upgrade AVRO version from 1.7.7 to 1.8.2 Sub-task Resolved Gengliang Wang
          3.
          		 support reading AVRO logical types - Date Sub-task Resolved Gengliang Wang
          4.
          		 support reading AVRO logical types - Timestamp with different precisions Sub-task Resolved Gengliang Wang
          5.
          		 support reading AVRO logical types - Decimal Sub-task Resolved Gengliang Wang
          6.
          		 support reading AVRO logical types - Duration Sub-task Resolved Unassigned
          7.
          		 AVRO unit test: use SQLTestUtils and Replace deprecated methods Sub-task Resolved Gengliang Wang
          8.
          		 Add write benchmark for AVRO Sub-task Resolved Gengliang Wang
          9.
          		 Add API `.avro` in DataFrameReader/DataFrameWriter Sub-task Resolved Unassigned
          10.
          		 Refactor Avro Serializer and Deserializer Sub-task Resolved Gengliang Wang
          11.
          		 Don't ignore files without .avro extension by default Sub-task Resolved Max Gekk
          12.
          		 Fix paths to resource files in AvroSuite Sub-task Resolved Max Gekk
          13.
          		 Support for parsing AVRO binary column Sub-task Resolved Unassigned
          14.
          		 Supporting to convert a column into binary of AVRO format Sub-task Resolved Unassigned
          15.
          		 New option - ignoreExtension Sub-task Resolved Max Gekk
          16.
          		 Gather all options into AvroOptions Sub-task Resolved Max Gekk
          17.
          		 Simplify schema serialization Sub-task Resolved Gengliang Wang
          18.
          		 New options - compression and compressionLevel Sub-task Resolved Max Gekk
          19.
          		 Remove implicit class AvroDataFrameWriter/AvroDataFrameReader Sub-task Resolved Gengliang Wang
          20.
          		 Use SerializableConfiguration in Spark util Sub-task Resolved Gengliang Wang
          21.
          		 Add mapping for built-in Avro data source Sub-task Resolved Dongjoon Hyun
          22.
          		 Use internal.Logging instead for logging Sub-task Resolved Hyukjin Kwon
          23.
          		 Avro: revise the output record namespace Sub-task Resolved Gengliang Wang
          24.
          		 Generate Avro Binary files in test suite Sub-task Resolved Gengliang Wang
          25.
          		 Validate user specified output schema Sub-task Resolved Gengliang Wang
          26.
          		 Make the mapping of com.databricks.spark.avro to built-in module configurable Sub-task Resolved Unassigned
          27.
          		 Documentaion: AVRO data source guide Sub-task Resolved Gengliang Wang
          28.
          		 Remove sql configuration spark.sql.avro.outputTimestampType Sub-task Resolved Gengliang Wang
          29.
          		 Detect recursive reference in Avro schema and throw exception Sub-task Resolved Gengliang Wang
          30.
          		 Support parse mode option for function `from_avro` Sub-task Resolved Gengliang Wang
          31.
          		 Override method `prettyName` in `from_avro`/`to_avro` Sub-task Resolved Gengliang Wang
          32.
          		 Add read benchmark for Avro Sub-task Resolved Gengliang Wang
          33.
          		 Avro: Validate input and output schema Sub-task Resolved Gengliang Wang
          34.
          		 Allow user-specified output schema in function `to_avro` Sub-task Resolved Gengliang Wang
          35.
          		 Show Avro related API in documentation Sub-task Resolved Gengliang Wang

          Activity

            People

              Gengliang.Wang Gengliang Wang
              Gengliang.Wang Gengliang Wang
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: