Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33354 New explicit cast syntax rules in ANSI mode
  3. SPARK-37714

ANSI mode: allow casting between numeric type and timestamp type

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      What changes were proposed?

      • By default, allow casting between numeric type and timestamp type under ANSI mode
      • Remove the user-facing configuration spark.sql.ansi.allowCastBetweenDatetimeAndNumeric

      Why are the changes needed?

      Same reason as mentioned in #34459. It is for better adoption of ANSI SQL mode since users are relying on it:

      • As we did some data science, we found that many Spark SQL users are actually using Cast(Timestamp as Numeric) and Cast(Numeric as Timestamp).
      • The Spark SQL connector for Tableau is using this feature for DateTime math. e.g.
        CAST(FROM_UNIXTIME(CAST(CAST(%1 AS BIGINT) + (%2 * 86400) AS BIGINT)) AS TIMESTAMP)

      Attachments

        Activity

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: