Details
-
Sub-task
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.0.0
Description
By default, collect() returns java.sql.Timestamp/Date instances with offsets derived from internal values of Catalyst's TIMESTAMP/DATE that store microseconds since the epoch. The conversion from internal values to java.sql.Timestamp/Date based on Proleptic Gregorian calendar but converting the resulted values before 1582 year to strings produces timestamp/date string in Julian calendar. For example:
scala> sql("select date '1100-10-10'").collect()
res1: Array[org.apache.spark.sql.Row] = Array([1100-10-03])
This can be fixed if internal Catalyst's values are converted to local date-time in Gregorian calendar, and construct local date-time from the resulted year, month, ..., seconds in Julian calendar.