[SPARK-24554] Add MapType Support for Arrow in PySpark

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 3.1.0
• Component/s: PySpark, SQL
• Labels: bulk-closed
• Assignee: Bryan Cutler
• Reporter: Bryan Cutler
• Created: 13/Jun/18 21:34
• Updated: 12/Dec/22 18:11
• Resolved: 18/Nov/20 12:19

Description

Add support for MapType in Arrow related classes in Scala/Java and pyarrow functionality in Python.

Issue Links

• depends upon: SPARK-29493 Add MapType support for Arrow Java
• depends upon: ARROW-6904 [Python] Implement MapArray and MapType
• depends upon: ARROW-10151 [Python] Add support MapArray to_pandas conversion
• is blocked by: ARROW-1279 [Integration][Java] Integration tests for Map type
• relates to: ARROW-3850 [Python] Support MapType and StructType for enhanced PySpark integration
• links to: [Github] Pull Request #30393 (BryanCutler)