[SPARK-18241] If Spark Launcher fails to startApplication then handle's state does not change

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.0.0
• Component/s: Spark Submit
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Aseem Bansal
• Created: 03/Nov/16 06:56
• Updated: 21/May/19 04:34
• Resolved: 21/May/19 04:34

Description

I am using Spark 2.0.0. I am using https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/launcher/SparkLauncher.html to submit my job.

If there is a failure after launcher's startapplication has been called but before the spark job has actually started (i.e. in starting the spark process that submits the job itself) there is

• no exception in the main thread that is submitting the job
• no exception in the job as it has not started
• no state change of the launcher
• the exception is logged in the error stream on the default logger name that spark produces using the Job's main class.

Basically, it is not possible to catch an exception if it happens during that time. The easiest way to reproduce it is to delete the JAR file or use an invalid spark home while launching the job using sparkLauncher.

Issue Links

• relates to: SPARK-17742 Spark Launcher does not get failed state in Listener