[SPARK-28041] Increase the minimum pandas version to 0.23.2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 13/Jun/19 17:15
• Updated: 12/Dec/22 18:10
• Resolved: 18/Jun/19 00:11

Description

Currently, the minimum supported Pandas version is 0.19.2. We bumped up testing in the Jenkins env to 0.23.2 and since 0.19.2 was released nearly 3 years ago, it is not always compatible with other Python libraries. Increasing the version to 0.23.2 will also allow some workarounds to be removed and make maintenance easier.

Issue Links

• is related to: SPARK-28003 spark.createDataFrame with Arrow doesn't work with pandas.NaT
• links to: GitHub Pull Request #24867