Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
3.0.1, 3.1.2, 3.2.0
-
None
-
None
Description
Please consider a following scenario on spark-3.0.1:
scala> List(("some date", new Date(Int.MaxValue)), ("some corner case date", new Date(Long.MaxValue))).toDF java.lang.RuntimeException: Error while encoding: java.lang.ArithmeticException: integer overflow staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._1, true, false) AS _1#0 staticinvoke(class org.apache.spark.sql.catalyst.util.DateTimeUtils$, DateType, fromJavaDate, knownnotnull(assertnotnull(input[0, scala.Tuple2, true]))._2, true, false) AS _2#1 at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:215) at org.apache.spark.sql.SparkSession.$anonfun$createDataset$1(SparkSession.scala:466) at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238) at scala.collection.immutable.List.foreach(List.scala:392) at scala.collection.TraversableLike.map(TraversableLike.scala:238) at scala.collection.TraversableLike.map$(TraversableLike.scala:231) at scala.collection.immutable.List.map(List.scala:298) at org.apache.spark.sql.SparkSession.createDataset(SparkSession.scala:466) at org.apache.spark.sql.SQLContext.createDataset(SQLContext.scala:353) at org.apache.spark.sql.SQLImplicits.localSeqToDatasetHolder(SQLImplicits.scala:231) ... 51 elided Caused by: java.lang.ArithmeticException: integer overflow at java.lang.Math.toIntExact(Math.java:1011) at org.apache.spark.sql.catalyst.util.DateTimeUtils$.fromJavaDate(DateTimeUtils.scala:111) at org.apache.spark.sql.catalyst.util.DateTimeUtils.fromJavaDate(DateTimeUtils.scala) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source) at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder$Serializer.apply(ExpressionEncoder.scala:211) ... 60 more
In opposition to spark-2.4.7 where it is possible to create dataframe with such values:
scala> val df = List(("some date", new Date(Int.MaxValue)), ("some corner case date", new Date(Long.MaxValue))).toDF df: org.apache.spark.sql.DataFrame = [_1: string, _2: date]scala> df.show +--------------------+-------------+ | _1| _2| +--------------------+-------------+ | some date| 1970-01-25| |some corner case ...|1701498-03-18| +--------------------+-------------+
Anyway, I am aware of the fact that during collecting these data I will got another result:
scala> df.collect
res10: Array[org.apache.spark.sql.Row] = Array([some date,1970-01-25], [some corner case date,?498-03-18])
what seems to be natural because of behaviour of java.sql.Date:
scala> new java.sql.Date(Long.MaxValue) res1: java.sql.Date = ?994-08-17
When it comes to easier reproduction, please consider:
scala> org.apache.spark.sql.catalyst.util.DateTimeUtils.fromJavaDate(new java.sql.Date(Long.MaxValue)) java.lang.ArithmeticException: integer overflow at java.lang.Math.toIntExact(Math.java:1011) at org.apache.spark.sql.catalyst.util.DateTimeUtils$.fromJavaDate(DateTimeUtils.scala:111) ... 47 elided
However, the question is even if such late dates are not supported, could it fail in more gentle way?