[SPARK-12128] Multiplication on decimals in dataframe returns null

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 1.5.0, 1.5.1, 1.5.2
• Component/s: Spark Core

Environment

Scala 2.11/Spark 1.5.0/1.5.1/1.5.2

• Assignee: Unassigned
• Reporter: Philip Dodds
• Created: 03/Dec/15 13:58
• Updated: 04/Dec/15 19:19
• Resolved: 03/Dec/15 17:20

Description

I hit a weird issue when I tried to multiply to decimals in a select (either in scala or as SQL), and Im assuming I must be missing the point.

The issue is fairly easy to recreate with something like the following:

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._
import org.apache.spark.sql.types.Decimal

case class Trade(quantity: Decimal,price: Decimal)

val data = Seq.fill(100) {
  val price = Decimal(20+scala.util.Random.nextInt(10))
    val quantity = Decimal(20+scala.util.Random.nextInt(10))

  Trade(quantity, price)
}

val trades = sc.parallelize(data).toDF()
trades.registerTempTable("trades")

trades.select(trades("price")*trades("quantity")).show

sqlContext.sql("select price/quantity,price*quantity,price+quantity,price-quantity from trades").show

The odd part is if you run it you will see that the addition/division and subtraction works but the multiplication returns a null.

Tested on 1.5.1/1.5.2 (Scala 2.10 and 2.11)

ie.

+------------------+

|(price * quantity)|

+------------------+

|              null|

|              null|

|              null|

|              null|

|              null|

+------------------+



+--------------------+----+--------------------+--------------------+

|                 _c0| _c1|                 _c2|                 _c3|

+--------------------+----+--------------------+--------------------+

|0.952380952380952381|null|41.00000000000000...|-1.00000000000000...|

|1.380952380952380952|null|50.00000000000000...|8.000000000000000000|

|1.272727272727272727|null|50.00000000000000...|6.000000000000000000|

|0.833333333333333333|null|44.00000000000000...|-4.00000000000000...|

|1.000000000000000000|null|58.00000000000000...|               0E-18|

+--------------------+----+--------------------+--------------------+