[SPARK-23754] StopIterator exception in Python UDF results in partial result

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: PySpark
• Target Version/s: 2.3.1
• Assignee: Emilio Dorigatti
• Reporter: Li Jin
• Created: 20/Mar/18 19:09
• Updated: 12/Dec/22 18:11
• Resolved: 30/May/18 10:14

Description

Reproduce:

df = spark.range(0, 1000)
from pyspark.sql.functions import udf

def foo(x):
    raise StopIteration()

df.withColumn('v', udf(foo)).show()

# Results
# +---+---+
# | id|  v|
# +---+---+
# +---+---+

I think the task should fail in this case

Issue Links

• is duplicated by: SPARK-24034 StopIteration in pyspark mapper gives partial results
• links to: [Github] Pull Request #21383 (e-dorigatti)
• links to: [Github] Pull Request #21461 (e-dorigatti)
• links to: [Github] Pull Request #21463 (e-dorigatti)
• links to: [Github] Pull Request #21467 (e-dorigatti)
• links to: [Github] Pull Request #21538 (e-dorigatti)