[SPARK-40121] Initialize projection used for Python UDF

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3, 3.3.0, 3.2.2, 3.4.0
• Fix Version/s: 3.1.4, 3.3.1, 3.2.3, 3.4.0
• Component/s: PySpark, SQL
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 17/Aug/22 10:01
• Updated: 12/Dec/22 18:10
• Resolved: 18/Aug/22 03:28

Description

>>> from pyspark.sql.functions import udf, rand
>>> spark.range(10).select(udf(lambda x: x)(rand())).show()

java.lang.NullPointerException
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificMutableProjection.apply(Unknown Source)
	at org.apache.spark.sql.execution.python.EvalPythonExec.$anonfun$doExecute$10(EvalPythonExec.scala:126)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$GroupedIterator.takeDestructively(Iterator.scala:1161)
	at scala.collection.Iterator$GroupedIterator.go(Iterator.scala:1176)
	at scala.collection.Iterator$GroupedIterator.fill(Iterator.scala:1213)

Issue Links

• links to: [Github] Pull Request #37552 (HyukjinKwon)