[SPARK-22860] Spark workers log ssl passwords passed to the executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Jungtaek Lim
• Reporter: Felix K.
• Created: 21/Dec/17 15:50
• Updated: 05/Dec/19 23:51
• Resolved: 26/Feb/19 22:50

Description

The workers log the spark.ssl.keyStorePassword and spark.ssl.trustStorePassword passed by cli to the executor processes. The ExecutorRunner should escape passwords to not appear in the worker's log files in INFO level. In this example, you can see my 'SuperSecretPassword' in a worker log:

17/12/08 08:04:12 INFO ExecutorRunner: Launch command: "/global/myapp/oem/jdk/bin/java" "-cp" "/global/myapp/application/myapp_software/thing_loader_lib/core-repository-model-zzz-1.2.3-SNAPSHOT.jar
[...]
:/global/myapp/application/spark-2.1.1-bin-hadoop2.7/jars/*" "-Xmx16384M" "-Dspark.authenticate.enableSaslEncryption=true" "-Dspark.ssl.keyStorePassword=SuperSecretPassword" "-Dspark.ssl.keyStore=/global/myapp/application/config/ssl/keystore.jks" "-Dspark.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks" "-Dspark.ssl.enabled=true" "-Dspark.driver.port=39927" "-Dspark.ssl.protocol=TLS" "-Dspark.ssl.trustStorePassword=SuperSecretPassword" "-Dspark.authenticate=true" "-Dmyapp_IMPORT_DATE=2017-10-30" "-Dmyapp.config.directory=/global/myapp/application/config" "-Dsolr.httpclient.builder.factory=com.company.myapp.loader.auth.LoaderConfigSparkSolrBasicAuthConfigurer" "-Djavax.net.ssl.trustStore=/global/myapp/application/config/ssl/truststore.jks" "-XX:+UseG1GC" "-XX:+UseStringDeduplication" "-Dthings.loader.export.zzz_files=false" "-Dlog4j.configuration=file:/global/myapp/application/config/spark-executor-log4j.properties" "-XX:+HeapDumpOnOutOfMemoryError" "-XX:+UseStringDeduplication" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@192.168.0.1:39927" "--executor-id" "2" "--hostname" "192.168.0.1" "--cores" "4" "--app-id" "app-20171208080412-0000" "--worker-url" "spark://Worker@192.168.0.1:59530"

Issue Links

• is related to: SPARK-26998 spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode
• links to: [Github] Pull Request #21514 (tooptoop4)
• links to: [Github] Pull Request #23820 (HeartSaVioR)
• links to: [Github] Pull Request #23820 (HeartSaVioR)
• links to: GitHub Pull Request #23820