Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-23476

Spark will not start in local mode with authentication on

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: Spark Core, Spark Shell
    • Labels:
      None

      Description

      If spark is run with "spark.authenticate=true", then it will fail to start in local mode.

      17/02/03 12:09:39 ERROR spark.SparkContext: Error initializing SparkContext.
      java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
      	at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:401)
      	at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:221)
      	at org.apache.spark.SparkEnv$.create(SparkEnv.scala:258)
      	at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:199)
      	at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:290)
      ...
      

      It can be confusing when authentication is turned on by default in a cluster, and one tries to start spark in local mode for a simple test.

      Workaround: If spark.authenticate=true is specified as a cluster wide config, then the following has to be added
      --conf "spark.authenticate=false" --conf "spark.shuffle.service.enabled=false" --conf "spark.dynamicAllocation.enabled=false" --conf "spark.network.crypto.enabled=false" --conf "spark.authenticate.enableSaslEncryption=false"
      in the spark-submit command.

        Attachments

          Activity

            People

            • Assignee:
              gsomogyi Gabor Somogyi
              Reporter:
              gsomogyi Gabor Somogyi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: