Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24768

Have a built-in AVRO data source implementation

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

      Apache Avro (https://avro.apache.org) is a popular data serialization format. It is widely used in the Spark and Hadoop ecosystem, especially for Kafka-based data pipelines.  Using the external package https://github.com/databricks/spark-avro, Spark SQL can read and write the avro data. Making spark-Avro built-in can provide a better experience for first-time users of Spark SQL and structured streaming. We expect the built-in Avro data source can further improve the adoption of structured streaming. The proposal is to inline code from spark-avro package (https://github.com/databricks/spark-avro). The target release is Spark 2.4.  

      Attachments

        Issue Links

        1.
        Add function `from_avro` and `to_avro` Sub-task Resolved Gengliang Wang Actions
        2.
        Upgrade AVRO version from 1.7.7 to 1.8.2 Sub-task Resolved Gengliang Wang Actions
        3.
        support reading AVRO logical types - Date Sub-task Resolved Gengliang Wang Actions
        4.
        support reading AVRO logical types - Timestamp with different precisions Sub-task Resolved Gengliang Wang Actions
        5.
        support reading AVRO logical types - Decimal Sub-task Resolved Gengliang Wang Actions
        6.
        support reading AVRO logical types - Duration Sub-task Resolved Unassigned Actions
        7.
        AVRO unit test: use SQLTestUtils and Replace deprecated methods Sub-task Resolved Gengliang Wang Actions
        8.
        Add write benchmark for AVRO Sub-task Resolved Gengliang Wang Actions
        9.
        Add API `.avro` in DataFrameReader/DataFrameWriter Sub-task Resolved Unassigned Actions
        10.
        Refactor Avro Serializer and Deserializer Sub-task Resolved Gengliang Wang Actions
        11.
        Don't ignore files without .avro extension by default Sub-task Resolved Max Gekk Actions
        12.
        Fix paths to resource files in AvroSuite Sub-task Resolved Max Gekk Actions
        13.
        Support for parsing AVRO binary column Sub-task Resolved Unassigned Actions
        14.
        Supporting to convert a column into binary of AVRO format Sub-task Resolved Unassigned Actions
        15.
        New option - ignoreExtension Sub-task Resolved Max Gekk Actions
        16.
        Gather all options into AvroOptions Sub-task Resolved Max Gekk Actions
        17.
        Simplify schema serialization Sub-task Resolved Gengliang Wang Actions
        18.
        New options - compression and compressionLevel Sub-task Resolved Max Gekk Actions
        19.
        Remove implicit class AvroDataFrameWriter/AvroDataFrameReader Sub-task Resolved Gengliang Wang Actions
        20.
        Use SerializableConfiguration in Spark util Sub-task Resolved Gengliang Wang Actions
        21.
        Add mapping for built-in Avro data source Sub-task Resolved Dongjoon Hyun Actions
        22.
        Use internal.Logging instead for logging Sub-task Resolved Hyukjin Kwon Actions
        23.
        Avro: revise the output record namespace Sub-task Resolved Gengliang Wang Actions
        24.
        Generate Avro Binary files in test suite Sub-task Resolved Gengliang Wang Actions
        25.
        Validate user specified output schema Sub-task Resolved Gengliang Wang Actions
        26.
        Make the mapping of com.databricks.spark.avro to built-in module configurable Sub-task Resolved Unassigned Actions
        27.
        Documentaion: AVRO data source guide Sub-task Resolved Gengliang Wang Actions
        28.
        Remove sql configuration spark.sql.avro.outputTimestampType Sub-task Resolved Gengliang Wang Actions
        29.
        Detect recursive reference in Avro schema and throw exception Sub-task Resolved Gengliang Wang Actions
        30.
        Support parse mode option for function `from_avro` Sub-task Resolved Gengliang Wang Actions
        31.
        Override method `prettyName` in `from_avro`/`to_avro` Sub-task Resolved Gengliang Wang Actions
        32.
        Add read benchmark for Avro Sub-task Resolved Gengliang Wang Actions
        33.
        Avro: Validate input and output schema Sub-task Resolved Gengliang Wang Actions
        34.
        Allow user-specified output schema in function `to_avro` Sub-task Resolved Gengliang Wang Actions
        35.
        Show Avro related API in documentation Sub-task Resolved Gengliang Wang Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            12 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment