[ZEPPELIN-1511] Problem with Spark interpreter dependency loading using interpreter setting GUI

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Component/s: Interpreters

Environment

Environments

• Spark version: 2.0
• Scala version: 2.11
• Zeppelin version: 0.7.0-SNAPSHOT
• OS: MacOS (Sierra)
• Core: 8
• Driver Mem: 4G
• Not using embedded Spark. I have $SPARK_HOME
• My build options:

  ./dev/change_scala_version.sh 2.11
  mvn clean package -Pspark-2.0 -Phadoop-2.4 -Pyarn -Ppyspark -Psparkr -Pscala-2.11 -DskipTests
  

• Assignee: Unassigned
• Reporter: Tae Jun Kim
• Created: 30/Sep/16 09:45
• Updated: 18/Jul/18 07:23
• Resolved: 02/Oct/16 09:37

Description

Depenedency with problem

org.bitbucket.eunjeon:seunjeon_2.11:1.1.0 (Korean Morpheme Ananlyzer)
It's success if you can import org.bitbucket.eunjeon.seunjeon.Analyzer

Description

I'm trying to add the dependency with GUI(interpreter setting). It works just fine with com.databricks:spark-xml_2.11:0.4.0, but not with org.bitbucket.eunjeon:seunjeon_2.11:1.1.0
sc.version prints nothing. I will attach my screenshot.

However, it works with SPARK_SUBMIT_OPTIONS! Which is:

conf/zeppelin-env.sh

export SPARK_SUBMIT_OPTIONS="--packages org.bitbucket.eunjeon:seunjeon_2.11:1.1.0 --driver-memory 4G"

So, for now, I'm using this method.

I also added my Spark interpreter log below.

I saw codes of problem, but I don't know why T_T

Please help!

Spark Interpreter Log

zeppelin-interpreter-spark-xxx.log

INFO [2016-09-30 17:18:45,817] ({Thread-0} RemoteInterpreterServer.java[run]:90) - Starting remote interpreter server on port 54273
 INFO [2016-09-30 17:18:49,468] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter org.apache.zeppelin.spark.SparkInterpreter
 INFO [2016-09-30 17:18:49,488] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter org.apache.zeppelin.spark.SparkSqlInterpreter
 INFO [2016-09-30 17:18:49,491] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter org.apache.zeppelin.spark.DepInterpreter
 INFO [2016-09-30 17:18:49,495] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter org.apache.zeppelin.spark.PySparkInterpreter
 INFO [2016-09-30 17:18:49,496] ({pool-1-thread-2} RemoteInterpreterServer.java[createInterpreter]:186) - Instantiate interpreter org.apache.zeppelin.spark.SparkRInterpreter
 INFO [2016-09-30 17:18:49,524] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1475223529523 started by scheduler org.apache.zeppelin.spark.SparkInterpreter1779785062
ERROR [2016-09-30 17:18:49,988] ({pool-2-thread-2} Job.java[run]:189) - Job failed
java.lang.NoSuchMethodError: scala.reflect.internal.settings.MutableSettings$SettingValue.valueSetByUser()Lscala/Option;
    at scala.tools.nsc.Global.<init>(Global.scala:334)
    at scala.tools.nsc.interpreter.IMain$$anon$1.<init>(IMain.scala:247)
    at scala.tools.nsc.interpreter.IMain.newCompiler(IMain.scala:247)
    at scala.tools.nsc.interpreter.IMain.<init>(IMain.scala:93)
    at scala.tools.nsc.interpreter.IMain.<init>(IMain.scala:113)
    at scala.tools.nsc.interpreter.ILoop$ILoopInterpreter.<init>(ILoop.scala:108)
    at scala.tools.nsc.interpreter.ILoop.createInterpreter(ILoop.scala:118)
    at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:722)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:69)
    at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)
    at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:390)
    at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
    at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
 INFO [2016-09-30 17:18:49,993] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1475223529523 finished by scheduler org.apache.zeppelin.spark.SparkInterpreter1779785062

Issue Links

• is duplicated by: ZEPPELIN-3632 MutableSettings.YpartialUnification error