Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36062

Try to capture faulthanlder when a Python worker crashes.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • PySpark
    • None

    Description

      Currently, we just see an error message saying "exited unexpectedly (crashed)" when the UDFs causes the Python worker to crash by like segmentation fault.
      We should take advantage of faulthandler and try to capture the error message from the faulthandler.

      Attachments

        Activity

          People

            ueshin Takuya Ueshin
            ueshin Takuya Ueshin
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: