[SPARK-38098] Add support for ArrayType of nested StructType to arrow-based conversion

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.1
• Fix Version/s: 3.4.0
• Component/s: PySpark
• Labels: pull-request-available
• Assignee: Luca Canali
• Reporter: Luca Canali
• Created: 03/Feb/22 15:21
• Updated: 16/Apr/24 14:32
• Resolved: 22/Sep/22 17:10

Description

This proposes to add support for ArrayType of nested StructType to arrow-based conversion.
This allows Pandas UDFs, mapInArrow UDFs, and toPandas to operate on columns of type Array of Struct, via arrow serialization.

Issue Links

• links to: [Github] Pull Request #35391 (LucaCanali)
• links to: [Github] Pull Request #35391 (LucaCanali)
• links to: GitHub Pull Request #35391