XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently, Spark provides 2 SQL configs to control rebasing of dates/timestamps in Parquet and Avro datasource: 

      spark.sql.legacy.parquet.rebaseDateTime.enabled

      spark.sql.legacy.avro.rebaseDateTime.enabled

      The configs control rebasing in read and in write. That's can be inconvenient for users who want to read files saved by Spark 2.4 and earlier versions, and save dates/timestamps without rebasing.

      The ticket aims to split the configs, and introduce separate SQL configs for read and for write. 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                maxgekk Max Gekk
                Reporter:
                maxgekk Max Gekk
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: