Major Description
To reproduce:
import spark.implicits._ val ds = spark .createDataset(Seq(BigDecimal("1" * 20), BigDecimal("9" * 20))) .agg(sum("value")) .as[BigDecimal] ds.collect shouldEqual Seq(null)
Given the option to throw exception on overflow on, sum aggregation of overflowing bigdecimal still remain null. DecimalAggregates is only invoked when expression of the sum (not the elements to be operated) has sufficiently small precision. The fix seems to be in Sum expression itself.
Attachments
Issue Links
- links to