[ARROW-3850] [Python] Support MapType and StructType for enhanced PySpark integration

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.1
• Fix Version/s: 2.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/20364
• Assignee: Unassigned
• Reporter: Florian Wilhelm
• Created: 22/Nov/18 07:26
• Updated: 11/Jan/23 07:30
• Resolved: 20/Nov/20 19:47

Description

It would be great to support MapType and (nested) StructType in Arrow so that PySpark can make use of it.

 
Quite often as in my use-case in Hive table cells are also complex types saved. Currently it's not possible to user the new pandas_udf decorator which internally uses Arrow to generate a UDF for columns with complex types.

Issue Links

• is related to: SPARK-24554 Add MapType Support for Arrow in PySpark
• relates to: ARROW-6904 [Python] Implement MapArray and MapType