[SPARK-28003] spark.createDataFrame with Arrow doesn't work with pandas.NaT

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.3, 2.4.3
• Fix Version/s: 3.0.0
• Component/s: PySpark
• Assignee: Li Jin
• Reporter: Li Jin
• Created: 11/Jun/19 17:32
• Updated: 24/Jun/19 20:23
• Resolved: 24/Jun/19 18:15

Description

import pandas as pd
dt1 = [pd.NaT, pd.Timestamp('2019-06-11')] * 100
pdf1 = pd.DataFrame({'time': dt1})

df1 = self.spark.createDataFrame(pdf1)

The example above doesn't work with arrow enabled.

Issue Links

• relates to: SPARK-28041 Increase the minimum pandas version to 0.23.2
• links to: GitHub Pull Request #24844