Description
If you create a Dataset[scala.math.BigInt], then you can't have a precision > 19.
scala> case class BigIntWrapper(value:scala.math.BigInt) defined class BigIntWrapper scala> val longDf = spark.createDataset(BigIntWrapper(scala.math.BigInt("10000000000000000002"))::Nil) 17/04/14 19:45:15 main INFO CodeGenerator: Code generated in 211.949738 ms java.lang.RuntimeException: Error while encoding: java.lang.IllegalArgumentException: requirement failed: BigInteger 10000000000000000002 too large for decimal staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), apply, assertnotnull(input[0, BigIntWrapper, true], top level non-flat input object).value, true) AS value#16 +- staticinvoke(class org.apache.spark.sql.types.Decimal$, DecimalType(38,0), apply, assertnotnull(input[0, BigIntWrapper, true], top level non-flat input object).value, true) +- assertnotnull(input[0, BigIntWrapper, true], top level non-flat input object).value +- assertnotnull(input[0, BigIntWrapper, true], top level non-flat input object) +- input[0, BigIntWrapper, true] at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:280) at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421) at org.apache.spark.sql.SparkSession$$anonfun$3.apply(SparkSession.scala:421) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.immutable.List.map(List.scala:285) at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:421) ... 54 elided Caused by: java.lang.IllegalArgumentException: requirement failed: BigInteger 10000000000000000002 too large for decimal at scala.Predef$.require(Predef.scala:224) at org.apache.spark.sql.types.Decimal.set(Decimal.scala:137) at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:419) at org.apache.spark.sql.types.Decimal.apply(Decimal.scala) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:277) ... 62 more
Attachments
Issue Links
- links to