Details
-
Wish
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.2.1
-
None
Description
When '-verbose' is passed to SparkSubmit it prints some useful debug information: Main class, Argument and Spark config.
I find it a bit hard to find information there because the arguments/configs are shuffled. I suggest to sort those before printing them.
Main class: org.apache.spark.deploy.k8s.submit.KubernetesClientApplication Arguments: --main-class --primary-java-resource local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar org.apache.spark.examples.SparkPi Spark config: (spark.app.name,spark-on-k8s-app) (spark.app.submitTime,1645106476125) (spark.driver.cores,1) (spark.driver.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true) (spark.driver.memory,2048m) (spark.dynamicAllocation.enabled,true) (spark.dynamicAllocation.shuffleTracking.enabled,true) (spark.executor.cores,2) (spark.executor.extraJavaOptions,-Dio.netty.tryReflectionSetAccessible=true) (spark.executor.instances,3) (spark.executor.memory,2048m) (spark.jars,local:///opt/spark/examples/jars/spark-examples_2.13-3.3.0-SNAPSHOT.jar) (spark.kubernetes.allocation.batch.delay,1) (spark.kubernetes.allocation.batch.size,3) (spark.kubernetes.authenticate.driver.serviceAccountName,spark-account-name) (spark.kubernetes.driver.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim) (spark.kubernetes.executor.container.image,spark/spark:3.3.0-SNAPSHOT-scala_2.13-11-jre-slim) (spark.kubernetes.namespace,spark-on-k8s) (spark.master,k8s://https://192.168.49.2:8443) (spark.network.timeout,300) (spark.submit.deployMode,cluster) (spark.submit.pyFiles,) Classpath elements:
The "Parsed arguments:" order is hardcoded at org.apache.spark.deploy.SparkSubmitArguments#toString, so they are still shuffled.
