[SPARK-31963] Support both pandas 0.23 and 1.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: PySpark, SQL
• Assignee: William Hyun
• Reporter: William Hyun
• Created: 10/Jun/20 20:54
• Updated: 11/Jun/20 00:04
• Resolved: 10/Jun/20 21:43

Description

This issue aims to fix a bug by supporting both pandas 0.23 and 1.0.

$ pip install pandas==0.23.2

$ python -c "import pandas.CategoricalDtype"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas.CategoricalDtype'

$ python -c "from pandas.api.types import CategoricalDtype"

Issue Links

• is caused by: SPARK-25351 Handle Pandas category type when converting from Python with Arrow
• relates to: SPARK-31964 Avoid Pandas import for CategoricalDtype with Arrow conversion
• links to: [Github] Pull Request #28789 (williamhyun)
• links to: [Github] Pull Request #28789 (williamhyun)