[SPARK-23476] Spark will not start in local mode with authentication on

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Spark Core, Spark Shell
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 21/Feb/18 09:22
• Updated: 22/Feb/18 20:08
• Resolved: 22/Feb/18 20:08

Description

If spark is run with "spark.authenticate=true", then it will fail to start in local mode.

17/02/03 12:09:39 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
	at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:401)
	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:258)
	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199)
	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:290)
...

It can be confusing when authentication is turned on by default in a cluster, and one tries to start spark in local mode for a simple test.

Workaround: If spark.authenticate=true is specified as a cluster wide config, then the following has to be added
--conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf "spark.authenticate.enableSaslEncryption=false"
in the spark-submit command.

Issue Links

• links to: [Github] Pull Request #20652 (gaborgsomogyi)