Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24768 Have a built-in AVRO data source implementation
  3. SPARK-25160

Remove sql configuration spark.sql.avro.outputTimestampType

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • SQL
    • None

    Description

      In the PR for supporting logical timestamp types https://github.com/apache/spark/pull/21935, a SQL configuration spark.sql.avro.outputTimestampType is added, so that user can specify the output timestamp precision they want.

      With PR https://github.com/apache/spark/pull/21847,  the output file can be written with user specified types. 

      So no need to have such configuration. Otherwise to make it consistent we need to add configuration for all the Catalyst types that can be converted into different Avro types. 

      Attachments

        Activity

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: