Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.3.0, 2.3.1
-
None
-
Tested with version 2.3.1 on OSX and 2.3.0 on Linux.
Description
The code snippet
import datetime df = spark.createDataFrame([(datetime.datetime(1901, 1, 1, 0, 0, 0),)], ["ts"]) print("collect:", df.collect()[0][0]) print("toPandas:", df.toPandas().iloc[0, 0])
prints
collect: 1901-01-01 00:00:00 toPandas: 1900-12-31 23:53:00
Hence the toPandas methods seems to convert the timestamp wrongly.
The problem disappears for "1902-01-01 00:00:00" and later dates (I didn't do an exhaustive test though).