[SPARK-43412] Introduce `SQL_ARROW_BATCHED_UDF` EvalType for Arrow Python UDFs

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 3.5.0
• Component/s: Connect, PySpark
• Assignee: Xinrong Meng
• Reporter: Xinrong Meng
• Created: 08/May/23 18:34
• Updated: 14/Jun/23 21:53
• Resolved: 10/May/23 20:10

Description

We are about to improve nested non-atomic input/output support of an Arrow-optimized Python UDF.

However, currently, it shares the same EvalType with a pickled Python UDF, but the same implementation with a Pandas UDF.

Introducing an EvalType enables isolating the changes to Arrow-optimized Python UDFs.

Issue Links

• links to: [Github] Pull Request #41053 (xinrong-meng)