[SPARK-49197] Redact `Spark Command` output in `launcher` module

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.2, 3.4.3
• Fix Version/s: 4.0.0, 3.4.4, 3.5.3
• Component/s: Spark Core
• Labels: pull-request-available
• Assignee: Dongjoon Hyun
• Reporter: Dongjoon Hyun
• Created: 12/Aug/24 01:54
• Updated: 12/Aug/24 20:05
• Resolved: 12/Aug/24 17:29

Description

When `launcher` module shows `Spark Command`, there is no redaction. Although Spark Cluster is supposed to be in a secure environment, this could be collected by a centralized log system. We need to do a proper redaction like we do for `Launch Command:` output.

$ SPARK_NO_DAEMONIZE=1 SPARK_MASTER_OPTS="-Dspark.master.rest.enabled=true -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4=" sbin/start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/logs/spark-dongjoon-org.apache.spark.deploy.master.Master-1-M3-Max.local.out
Spark Command: /Users/dongjoon/.jenv/versions/17/bin/java -cp /Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/conf/:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/slf4j-api-2.0.13.jar:/Users/dongjoon/APACHE/spark-releases/spark-4.0.0-preview1-bin-hadoop3/jars/* -Dspark.master.rest.enabled=true -Dspark.master.rest.filters=org.apache.spark.ui.JWSFilter -Dspark.org.apache.spark.ui.JWSFilter.param.secretKey=VmlzaXQgaHR0cHM6Ly9zcGFyay5hcGFjaGUub3JnIHRvIGRvd25sb2FkIEFwYWNoZSBTcGFyay4= -Xmx1g org.apache.spark.deploy.master.Master --host M3-Max.local --port 7077 --webui-port 8080

Issue Links

• links to: GitHub Pull Request #47704