[SPARK-35955] Fix decimal overflow issues for Average

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Karen Feng
• Reporter: Karen Feng
• Created: 30/Jun/21 17:27
• Updated: 02/Jul/21 04:41
• Resolved: 02/Jul/21 04:41

Description

Fix decimal overflow issues for decimal average in ANSI mode. Linked to SPARK-32018 and SPARK-28067, which address decimal sum.

Repro:

 

import org.apache.spark.sql.functions._
spark.conf.set("spark.sql.ansi.enabled", true)

val df = Seq(
 (BigDecimal("10000000000000000000"), 1),
 (BigDecimal("10000000000000000000"), 1),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2),
 (BigDecimal("10000000000000000000"), 2)).toDF("decNum", "intNum")
val df2 = df.withColumnRenamed("decNum", "decNum2").join(df, "intNum").agg(mean("decNum"))
df2.show(40,false)

 

Should throw an exception (as sum overflows), but instead returns:

 

+-----------+
|avg(decNum)|
+-----------+
|null       |
+-----------+

 

Issue Links

• links to: [Github] Pull Request #33177 (karenfeng)