Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44763

Fix a bug of promoting string as double in binary arithmetic with interval

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.0.0
    • None
    • SQL
    • None

    Description

      The following query works on branch-3.5 or below, but fails on the latest master:

      ```
      select concat(DATE'2020-12-31', ' ', date_format('09:03:08', 'HH:mm:ss')) + (INTERVAL '03' HOUR)
      ```
       
      The direct reason is now we mark `cast(date as string)` as resolved during type coercion after changes https://github.com/apache/spark/pull/42089. As a result, there are two transforms from CombinedTypeCoercionRule
      ```
      Rule ConcatCoercion Transformed concat(2020-12-31,  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) to concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles)))

      Rule PromoteStrings Transformed (concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) + INTERVAL '03' HOUR) to (cast(concat(cast(2020-12-31 as string),  , date_format(cast(09:03:08 as timestamp), HH:mm:ss, Some(America/Los_Angeles))) as double) + INTERVAL '03' HOUR)
      ```  
      The second transform doesn't happen in previous releases since cast(2020-12-31 as string)  used to be unresolved after the first transform.
       
      The fix is simple, the analyzer should not promote string as double in binary arithmetic with ANSI interval. The changes in https://github.com/apache/spark/pull/42089 are valid and we should keep it.
       

      Attachments

        Activity

          People

            Gengliang.Wang Gengliang Wang
            Gengliang.Wang Gengliang Wang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: