[SPARK-43323] DataFrame.toPandas with Arrow enabled should handle exceptions properly

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 29/Apr/23 01:32
• Updated: 01/May/23 23:17
• Resolved: 01/May/23 23:17

Description

Currently DataFrame.toPandas doesn't capture exceptions happened in Spark properly.

>>> spark.conf.set("spark.sql.ansi.enabled", True)
>>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> spark.sql("select 1/0").toPandas()
...
  An error occurred while calling o53.getResult.
: org.apache.spark.SparkException: Exception thrown in awaitResult:
	at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:322)
...