[SPARK-35524] Pass objects as parameters to SparkSQL UDFs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 3.1.1
• Component/s: PySpark, Spark Core, SQL
• Labels: UDF UDT spark spark-sql
• Assignee: Unassigned
• Reporter: Julian Shalaby
• Created: 26/May/21 02:10
• Updated: 12/Dec/22 18:10
• Resolved: 26/May/21 06:33

Description

You can pass class objects directly to UDFs using the UDF format:

df.select("*").filter(myFunc(classObj)(col("colName")))

but the format:

"""SELECT * FROM view WHERE myFunc(classObj, "colName")"""

or

"""SELECT * FROM view WHERE myFunc(classObj)("colName")"""

does not work. This would be a very useful feature to have, especially being that UDTs are being made public again in 3.2.0