[SPARK-43194] PySpark 3.4.0 cannot convert timestamp-typed objects to pandas with pandas 2.0

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark

Environment

In [4]: import pandas as pd

In [5]: pd.__version__
Out[5]: '2.0.0'

In [6]: import pyspark as ps

In [7]: ps.__version__
Out[7]: '3.4.0'

• Target Version/s: 4.0.0
• Assignee: Unassigned
• Reporter: Phillip Cloud
• Created: 19/Apr/23 13:23
• Updated: 04/Sep/23 02:12
• Resolved: 04/Sep/23 02:12

Description

In [1]: from pyspark.sql import SparkSession

In [2]: session = SparkSession.builder.appName("test").getOrCreate()
23/04/19 09:21:42 WARN Utils: Your hostname, albatross resolves to a loopback address: 127.0.0.2; using 192.168.1.170 instead (on interface enp5s0)
23/04/19 09:21:42 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
23/04/19 09:21:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

In [3]: session.sql("select now()").toPandas()

Results in:

...
TypeError: Casting to unit-less dtype 'datetime64' is not supported. Pass e.g. 'datetime64[ns]' instead.