Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33339

Pyspark application will hang due to non Exception

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.5, 3.0.0, 3.0.1
    • Fix Version/s: 2.4.8, 3.0.2, 3.1.0
    • Component/s: PySpark
    • Labels:
      None

      Description

      When a system.exit exception occurs during the process, the python worker exits abnormally, and then the executor task is still waiting for the worker for reading from socket, causing it to hang.
      The system.exit exception may be caused by the user's error code, but spark should at least throw an error to remind the user, not get stuck
      we can run a simple test to reproduce this case:

      from pyspark.sql import SparkSession
      
      def err(line):
        raise SystemExit
      
      spark = SparkSession.builder.appName("test").getOrCreate()
      spark.sparkContext.parallelize(range(1,2), 2).map(err).collect()
      spark.stop()
      

       

        Attachments

          Activity

            People

            • Assignee:
              lrz lrz
              Reporter:
              lrz lrz

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment