Critical Description
In Spark 3.3.0 SPARK-37475 PR[https://github.com/apache/spark/pull/34729] went in and added support for a scale parameter to floor and ceil. There was discussion about potential incompatibilities, specifically with respect to the return types. It looks like it was [decided|https://github.com/apache/spark/pull/34729#discussion_r767446855 to keep the old behavior if no scale parameter was passed in, but use the new functionality if a scale is passed in.
But the scala API didn't get updated to do the same thing as the SQL API.
scala> spark.range(1).selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id, 0) as two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"), ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema root |-- id: long (nullable = false) |-- one_arg_sql: long (nullable = true) |-- two_arg_sql: decimal(20,0) (nullable = true) |-- one_arg_func: decimal(20,0) (nullable = true) |-- two_arg_func: decimal(20,0) (nullable = true) scala> spark.range(1).selectExpr("cast(id as double) as id").selectExpr("id", "ceil(id) as one_arg_sql", "ceil(id, 0) as two_arg_sql").select(col("*"), ceil(col("id")).alias("one_arg_func"), ceil(col("id"), lit(0)).alias("two_arg_func")).printSchema root |-- id: double (nullable = false) |-- one_arg_sql: long (nullable = true) |-- two_arg_sql: decimal(30,0) (nullable = true) |-- one_arg_func: decimal(30,0) (nullable = true) |-- two_arg_func: decimal(30,0) (nullable = true)
And because the python code call into this too it also has the same problem. I suspect that the java and R code also expose it too, but I didn't check.
