[SPARK-34600] Support user defined types in Pandas UDF

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.2, 3.1.1
• Component/s: PySpark, SQL
• Labels: pandas sql udf
• Assignee: Unassigned
• Reporter: Lei (Eddy) Xu
• Created: 02/Mar/21 23:47
• Updated: 12/Dec/22 18:10

Description

This is an umbrella ticket.

Issue Links

• links to: [Github] Pull Request #31735 (eddyxu)

Sub-Tasks

• Support UDT for Pandas/Spark conversion with Arrow support Enabled: In Progress
• Return UDT from Pandas UDF: @pandas_udf(UDT): In Progress
• Support TimestampType in UDT: Open
• Bug when creating dataframe without schema and with Arrow disabled: In Progress
• Bug when creating dataframe without schema and with Arrow disabled: Resolved