[SPARK-29070] Make SparkLauncher log full spark-submit command line

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.5
• Fix Version/s: 3.0.0
• Component/s: Spark Submit
• Assignee: Jeff Evans
• Reporter: Jeff Evans
• Created: 12/Sep/19 16:02
• Updated: 27/Sep/19 18:34
• Resolved: 27/Sep/19 18:34

Description

org.apache.spark.launcher.SparkLauncher wraps a ProcessBuilder, and builds up a full command line to spark-submit using a builder pattern. When startApplication is finally called, a full command line is materialized out of all the options, then invoked via the ProcessBuilder.

In scenarios where another application is submitting to Spark, it would be extremely useful from a support and debugging standpoint to be able to see the full spark-submit command that is actually used (so that the same submission can be tested standalone, arguments tweaked, etc.). Currently, the only way this gets captured is to stderr if the SPARK_PRINT_LAUNCH_COMMAND environment variable is set. This is cumbersome in the context of an application that is wrapping Spark and already using the APIs.

I propose simply making SparkSubmit log the full command line it is about to launch, so that clients can see it directly in their log files, rather than having to capture and search through stderr.

Issue Links

• links to: GitHub Pull Request #25777