Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34246 New type coercion syntax rules in ANSI mode
  3. SPARK-38176

ANSI mode: allow implicitly casting String to other simple types

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      Compared to the default behavior, the current ANSI type coercion rules don't allow the following cases:

      • comparing String with other simple types, e.g. date/timestamp/int ...
      • arithmetic op with String and other simple types
      • Union/Intersect/Except with String and other simple types
      • SQL function expects non-string types but got  string input
      • other SQL operators..

      The original purpose is to prevent potential String parsing errors under ANSI mode. However, after doing research among real-world Spark SQL queries, I find that many users are actually using String as Date/Timestamp/Numeric in their queries. 

       

      To make the migration to ANSI mode easier, I suggest removing this limitation. 

      Attachments

        Activity

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: