[SPARK-35614] Make the conversion to pandas data-type-based for ExtensionDtypes

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 02/Jun/21 21:12
• Updated: 21/Jun/21 20:20
• Resolved: 21/Jun/21 20:20

Description

The conversion from/to pandas includes logic for checking ExtensionDtypes data types and behaving accordingly.

That makes code hard to change or maintain.

We want to introduce the Ops class per ExtensionDtypes data type, and then make the conversion from/to pandas data-type-based for ExtensionDtypes.

Issue Links

• links to: [Github] Pull Request #32910 (xinrong-databricks)