Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38604

ceil and floor return different types when called from scala than sql

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.3.0
    • 3.3.0
    • SQL
    • None

    Description

      In Spark 3.3.0  SPARK-37475 PR[https://github.com/apache/spark/pull/34729] went in and added support for a scale parameter to floor and ceil.  There was discussion about potential incompatibilities, specifically with respect to the return types. It looks like it was [decided|https://github.com/apache/spark/pull/34729#discussion_r767446855 to keep the old behavior if no scale parameter was passed in, but use the new functionality if a scale is passed in.

       

      But the scala API didn't get updated to do the same thing as the SQL API.

      scala> spark.range(1).selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id, 0) as two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"), ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema
      root
       |-- id: long (nullable = false)
       |-- one_arg_sql: long (nullable = true)
       |-- two_arg_sql: decimal(20,0) (nullable = true)
       |-- one_arg_func: decimal(20,0) (nullable = true)
       |-- two_arg_func: decimal(20,0) (nullable = true)
       
      
      scala> spark.range(1).selectExpr("cast(id as double) as id").selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id, 0) as two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"), ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema
      root
       |-- id: double (nullable = false)
       |-- one_arg_sql: long (nullable = true)
       |-- two_arg_sql: decimal(30,0) (nullable = true)
       |-- one_arg_func: decimal(30,0) (nullable = true)
       |-- two_arg_func: decimal(30,0) (nullable = true) 

      And because the python code call into this too it also has the same problem. I suspect that the java and R code also expose it too, but I didn't check.

       

       

      Attachments

        Activity

          People

            revans2 Robert Joseph Evans
            revans2 Robert Joseph Evans
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: