Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.3.0
-
None
Description
If spark is run with "spark.authenticate=true", then it will fail to start in local mode.
17/02/03 12:09:39 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:401) at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:258) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:290) ...
It can be confusing when authentication is turned on by default in a cluster, and one tries to start spark in local mode for a simple test.
Workaround: If spark.authenticate=true is specified as a cluster wide config, then the following has to be added
--conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf "spark.authenticate.enableSaslEncryption=false"
in the spark-submit command.