Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29070

Make SparkLauncher log full spark-submit command line

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.5
    • Fix Version/s: 3.0.0
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      org.apache.spark.launcher.SparkLauncher wraps a ProcessBuilder, and builds up a full command line to spark-submit using a builder pattern. When startApplication is finally called, a full command line is materialized out of all the options, then invoked via the ProcessBuilder.

      In scenarios where another application is submitting to Spark, it would be extremely useful from a support and debugging standpoint to be able to see the full spark-submit command that is actually used (so that the same submission can be tested standalone, arguments tweaked, etc.). Currently, the only way this gets captured is to stderr if the SPARK_PRINT_LAUNCH_COMMAND environment variable is set. This is cumbersome in the context of an application that is wrapping Spark and already using the APIs.

      I propose simply making SparkSubmit log the full command line it is about to launch, so that clients can see it directly in their log files, rather than having to capture and search through stderr.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jeff.w.evans Jeff Evans
                Reporter:
                jeff.w.evans Jeff Evans
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified