Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-1242

Should set property SPARK_YARN_MODE and do login before creating any spark stuff

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.6.1, 0.7.0
    • Component/s: Interpreters
    • Labels:
      None

      Description

      We run zeppelin on spark when spark authentication is turned on, but got the following exception

       INFO [2016-07-28 00:35:32,845] ({pool-2-thread-2} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin
       INFO [2016-07-28 00:35:32,846] ({pool-2-thread-2} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin
       INFO [2016-07-28 00:35:32,908] ({pool-1-thread-3} Logging.scala[logInfo]:58) - Changing view acls to: zeppelin
       INFO [2016-07-28 00:35:32,908] ({pool-1-thread-3} Logging.scala[logInfo]:58) - Changing modify acls to: zeppelin
      ERROR [2016-07-28 00:35:32,909] ({pool-2-thread-2} Job.java[run]:189) - Job failed
      java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
              at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:397)
              at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:219)
              at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
              at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
              at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
              at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:566)
              at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
              at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
              at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
              at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
              at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      ERROR [2016-07-28 00:35:32,918] ({pool-1-thread-3} TThreadPoolServer.java[run]:296) - Error occurred during processing of message.
      java.lang.IllegalArgumentException: Error: a secret key must be specified via the spark.authenticate.secret config
              at org.apache.spark.SecurityManager.generateSecretKey(SecurityManager.scala:397)
              at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:219)
              at org.apache.spark.repl.SparkIMain.<init>(SparkIMain.scala:118)
              at org.apache.spark.repl.SparkILoop$SparkILoopInterpreter.<init>(SparkILoop.scala:187)
              at org.apache.spark.repl.SparkILoop.createInterpreter(SparkILoop.scala:217)
              at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:566)
              at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
              at org.apache.zeppelin.interpreter.LazyOpenInterpreter.getProgress(LazyOpenInterpreter.java:110)
              at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getProgress(RemoteInterpreterServer.java:404)
              at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1509)
              at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getProgress.getResult(RemoteInterpreterService.java:1494)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
              at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      

      The root cause is that we didn't set property SPARK_YARN_MODE just like spark-shell did.
      The following code in Main.scala is what we miss in zeppelin spark interpreter. https://github.com/apache/spark/blob/branch-1.6/repl/scala-2.11/src/main/scala/org/apache/spark/repl/Main.scala

        def main(args: Array[String]) {
          if (getMaster == "yarn-client") System.setProperty("SPARK_YARN_MODE", "true")
      

      Thanks Kshitij Badani for finding this issue.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zjffdu Jeff Zhang
                Reporter:
                zjffdu Jeff Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: