[SPARK-42393] Support for Pandas/Arrow Functions API

• Type: Umbrella
• Status: Resolved
• Priority: Major
• Resolution: Resolved
• Affects Version/s: 3.4.0
• Component/s: Connect, PySpark
• Epic Link: Spark Connect
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 10/Feb/23 07:55
• Updated: 29/Mar/23 03:41
• Resolved: 28/Mar/23 09:55

Description

There are derivative APIs which depend on the implementation of Pandas UDFs: Pandas Function APIs and Arrow Function APIs, as shown below:

 

Spark Connect Python Client (SCPC), as a client and server interface for PySpark will eventually replace the legacy API of PySpark. Supporting PySpark UDFs is essential for Spark Connect to reach parity with the PySpark legacy API.

See design doc here.

Issue Links

• depends upon: SPARK-41661 Support for User-defined Functions in Python