Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24768

Have a built-in AVRO data source implementation

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines.  Using the external package https://github.com/databricks/spark-avro, Spark SQL can read and write the avro data. Making spark-Avro built-in can provide a better experience for first-time users of Spark SQL and structured streaming. We expect the built-in Avro data source can further improve the adoption of structured streaming. The proposal is to inline code from spark-avro package (https://github.com/databricks/spark-avro). The target release is Spark 2.4.  

        Attachments

          Issue Links

          1.
          Add function `from_avro` and `to_avro` Sub-task Resolved Gengliang Wang
          2.
          Upgrade AVRO version from 1.7.7 to 1.8.2 Sub-task Resolved Gengliang Wang
          3.
          support reading AVRO logical types - Date Sub-task Resolved Gengliang Wang
          4.
          support reading AVRO logical types - Timestamp with different precisions Sub-task Resolved Gengliang Wang
          5.
          support reading AVRO logical types - Decimal Sub-task Resolved Gengliang Wang
          6.
          support reading AVRO logical types - Duration Sub-task Resolved Unassigned
          7.
          AVRO unit test: use SQLTestUtils and Replace deprecated methods Sub-task Resolved Gengliang Wang
          8.
          Add write benchmark for AVRO Sub-task Resolved Gengliang Wang
          9.
          Add API `.avro` in DataFrameReader/DataFrameWriter Sub-task Resolved Unassigned
          10.
          Refactor Avro Serializer and Deserializer Sub-task Resolved Gengliang Wang
          11.
          Don't ignore files without .avro extension by default Sub-task Resolved Maxim Gekk
          12.
          Fix paths to resource files in AvroSuite Sub-task Resolved Maxim Gekk
          13.
          Support for parsing AVRO binary column Sub-task Resolved Unassigned
          14.
          Supporting to convert a column into binary of AVRO format Sub-task Resolved Unassigned
          15.
          New option - ignoreExtension Sub-task Resolved Maxim Gekk
          16.
          Gather all options into AvroOptions Sub-task Resolved Maxim Gekk
          17.
          Simplify schema serialization Sub-task Resolved Gengliang Wang
          18.
          New options - compression and compressionLevel Sub-task Resolved Maxim Gekk
          19.
          Remove implicit class AvroDataFrameWriter/AvroDataFrameReader Sub-task Resolved Gengliang Wang
          20.
          Use SerializableConfiguration in Spark util Sub-task Resolved Gengliang Wang
          21.
          Add mapping for built-in Avro data source Sub-task Resolved Dongjoon Hyun
          22.
          Use internal.Logging instead for logging Sub-task Resolved Hyukjin Kwon
          23.
          Avro: revise the output record namespace Sub-task Resolved Gengliang Wang
          24.
          Generate Avro Binary files in test suite Sub-task Resolved Gengliang Wang
          25.
          Validate user specified output schema Sub-task Resolved Gengliang Wang
          26.
          Make the mapping of com.databricks.spark.avro to built-in module configurable Sub-task Resolved Unassigned
          27.
          Documentaion: AVRO data source guide Sub-task Resolved Gengliang Wang
          28.
          Remove sql configuration spark.sql.avro.outputTimestampType Sub-task Resolved Gengliang Wang
          29.
          Detect recursive reference in Avro schema and throw exception Sub-task Resolved Gengliang Wang
          30.
          Support parse mode option for function `from_avro` Sub-task Resolved Gengliang Wang
          31.
          Override method `prettyName` in `from_avro`/`to_avro` Sub-task Resolved Gengliang Wang
          32.
          Add read benchmark for Avro Sub-task Resolved Gengliang Wang
          33.
          Avro: Validate input and output schema Sub-task Resolved Gengliang Wang

            Activity

              People

              • Assignee:
                Gengliang.Wang Gengliang Wang
                Reporter:
                Gengliang.Wang Gengliang Wang
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: