[SPARK-28264] Revisiting Python / pandas UDF

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: PySpark, SQL
• Labels: release-notes
• Target Version/s: 3.0.0
• Epic Link: Project Zen
• Assignee: Hyukjin Kwon
• Reporter: Reynold Xin
• Created: 05/Jul/19 20:50
• Updated: 12/Dec/22 18:11
• Resolved: 22/Jan/20 06:33

Description

In the past two years, the pandas UDFs are perhaps the most important changes to Spark for Python data science. However, these functionalities have evolved organically, leading to some inconsistencies and confusions among users. This document revisits UDF definition and naming, as a result of discussions among Xiangrui, Li Jin, Hyukjin, and Reynold.

See document here: https://docs.google.com/document/d/10Pkl-rqygGao2xQf6sddt0b-4FYK4g8qr_bXLKTL65A/edit#

 New proposal: https://docs.google.com/document/d/1-kV0FS_LF2zvaRh_GhkV32Uqksm_Sq8SvnBBmRyxm30/edit?usp=sharing

Issue Links

• links to: GitHub Pull Request #27165