[SPARK-36062] Try to capture faulthanlder when a Python worker crashes.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: PySpark
• Assignee: Takuya Ueshin
• Reporter: Takuya Ueshin
• Created: 08/Jul/21 21:13
• Updated: 12/Dec/22 18:10
• Resolved: 09/Jul/21 02:31

Description

Currently, we just see an error message saying "exited unexpectedly (crashed)" when the UDFs causes the Python worker to crash by like segmentation fault.
We should take advantage of faulthandler and try to capture the error message from the faulthandler.

Issue Links

• links to: [Github] Pull Request #33273 (ueshin)