[SPARK-48706] Python UDF in higher order functions should not throw internal error

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 25/Jun/24 03:15
• Updated: 26/Jun/24 05:40
• Resolved: 26/Jun/24 05:40

Description

from pyspark.sql.functions import transform, udf, col, array
spark.range(1).select(transform(array("id"), lambda x: udf(lambda y: y)(x))).collect()

throws an internal error:

	at org.apache.spark.SparkException$.internalError(SparkException.scala:88)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:92)
	at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotEvaluateExpressionError(QueryExecutionErrors.scala:73)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval(Expression.scala:507)
	at org.apache.spark.sql.catalyst.expressions.Unevaluable.eval$(Expression.scala:506)