[SPARK-25652] Wrong datetime conversion between Java and Python

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: PySpark
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Maciej Bryński
• Created: 05/Oct/18 17:03
• Updated: 12/Dec/22 18:10
• Resolved: 25/May/21 01:40

Description

Hi,
I found strange behaviour of Spark when using datetime from night of changing date (in CET).

The data from MySQL is wrongly converted and as a result fold=1 is added.

Sample code

MySQL column has DATETIME type and value: "2017-10-29 02:01:44"

spark.read.jdbc(URL).select("time_column").collect()
[Row(start_time=datetime.datetime(2017, 10, 29, 2, 1, 44, fold=1))]

As a comparison same query done by sqlalchemy.

engine = create_engine(URL)
engine.execute("select time_column from table").fetchone()
(datetime.datetime(2017, 10, 29, 2, 1, 44),)

I'm using Python 3.6. Both MySQL server and server where I'm doing queries are in CET timezone.