Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10549

scala 2.11 spark on yarn with security - Repl doesn't work

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.1, 1.6.0
    • Component/s: Spark Shell
    • Labels:
      None

      Description

      Trying to run spark on secure yarn built with scala 2.11 results in failure when trying to launch the spark shell.

      ./bin/spark-shell --master yarn-client

      Exception in thread "main" java.lang.ExceptionInInitializerError
      at org.apache.spark.repl.Main.main(Main.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:483)
      at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
      at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
      at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
      Caused by: java.lang.Exception: Error: a secret key must be specified via the spark.authenticate.secret config
      at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:395)
      at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:218)
      at org.apache.spark.repl.Main$.<init>(Main.scala:38)
      at org.apache.spark.repl.Main$.<clinit>(Main.scala)

      The reason is because it create the SecurityManager before is sets the SPARK_YARN_MODE to true.

        Attachments

          Activity

            People

            • Assignee:
              tgraves Thomas Graves
              Reporter:
              tgraves Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: