[SPARK-45768] Make faulthandler a runtime configuration for Python execution in SQL

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: PySpark, SQL
• Labels: pull-request-available
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 02/Nov/23 07:15
• Updated: 03/Nov/23 00:33
• Resolved: 03/Nov/23 00:33

Description

faulthanlder feature within PySpark is really useful especially to debug an errors that regular Python interpreter cannot catch out of the box such as a segmentation fault errors, see also https://github.com/apache/spark/pull/43600. It would be very useful to convert this as a runtime configuration.

Issue Links

• links to: GitHub Pull Request #43635