Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38242

Sort the SparkSubmit debug output

    XMLWordPrintableJSON

Details

    • Wish
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.3.0
    • Spark Submit
    • None

    Description

      When '-verbose' is passed to SparkSubmit it prints some useful debug information: Main class, Argument and Spark config.

      I find it a bit hard to find information there because the arguments/configs are shuffled. I suggest to sort those before printing them.

       

       Main class:
      org.apache.spark.deploy.k8s.submit.KubernetesClientApplication
      Arguments:
      --main-class
      --primary-java-resource
      local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar
      org.apache.spark.examples.SparkPi
      Spark config:
      (spark.app.name,spark-on-k8s-app)
      (spark.app.submitTime,1645106476125)
      (spark.driver.cores,1)
      (spark.driver.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true)
      (spark.driver.memory,2048m)
      (spark.dynamicAllocation.enabled,true)
      (spark.dynamicAllocation.shuffleTracking.enabled,true)
      (spark.executor.cores,2)
      (spark.executor.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true)
      (spark.executor.instances,3)
      (spark.executor.memory,2048m)
      (spark.jars,local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar)
      (spark.kubernetes.allocation.batch.delay,1)
      (spark.kubernetes.allocation.batch.size,3)
      (spark.kubernetes.authenticate.driver.serviceAccountName,spark-account-name)
      (spark.kubernetes.driver.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim)
      (spark.kubernetes.executor.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim)
      (spark.kubernetes.namespace,spark-on-k8s)
      (spark.master,k8s://https://192.168.49.2:8443)
      (spark.network.timeout,300)
      (spark.submit.deployMode,cluster)
      (spark.submit.pyFiles,)
      Classpath elements:

       

      The "Parsed arguments:" order is hardcoded at org.apache.spark.deploy.SparkSubmitArguments#toString, so they are still shuffled.

      Attachments

        Activity

          People

            mgrigorov Martin Tzvetanov Grigorov
            mgrigorov Martin Tzvetanov Grigorov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: