[ZEPPELIN-5246] Zeppelin in cluster mode doesn't create spark submit

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: interpreter-setting, Interpreters, spark
• Assignee: Unassigned
• Reporter: Ruslan Fialkovsky
• Created: 10/Feb/21 15:45
• Updated: 12/Aug/21 15:28

Description

Hello. I'm trying  to configure zeppelin cluster mode and running spark on yarn.

This is my interpreter conf on the picture and it works in zeppelin single node case.

 

So, seems zeppelin starts SparkContext instead of  spark-submit in zeppelin cluster mode :

 INFO [2021-02-10 18:34:16,838] ({Thread-1034} ClusterInterpreterCheckThread.java[run]:51) - ClusterInterpreterCheckThread run() >>>                                                                      │
 INFO [2021-02-10 18:34:16,848] ({SchedulerFactory2} ProcessLauncher.java[transition]:109) - Process state is transitioned to LAUNCHED                                                                    │
 INFO [2021-02-10 18:34:16,848] ({SchedulerFactory2} ProcessLauncher.java[launch]:96) - Process is launched: [/usr/lib/zeppelin/bin/interpreter.sh, -d, /usr/lib/zeppelin/interpreter/spark, -c, 10.15.145│
.26, -p, 17317, -r, :, -i, spark-fialkovskiy, -u, fialkovskiy, -l, /usr/lib/zeppelin/local-repo/spark, -g, spark]                                                                                         │
 INFO [2021-02-10 18:34:16,955] ({Exec Stream Pumper} ProcessLauncher.java[processLine]:188) - Interpreter launch command:  /usr/lib/spark/3.0.1/bin/spark-submit --class org.apache.zeppelin.interpreter.│
remote.RemoteInterpreterServer --driver-class-path ":/usr/lib/zeppelin/interpreter/spark/*::/usr/lib/zeppelin/interpreter/zeppelin-interpreter-shaded-0.9.0-preview2.jar:/usr/lib/zeppelin/interpreter/spa│
rk/spark-interpreter-0.9.0-preview2.jar:/etc/hadoop/" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration='file:///etc/zeppelin/log4j.properties' -Dlog4j.configurationFile='file:///etc/z│
eppelin/log4j2.properties' -Dzeppelin.log.file='/usr/lib/zeppelin/logs/zeppelin-interpreter-spark-fialkovskiy-fialkovskiy--hadoop836713.log'" /usr/lib/zeppelin/interpreter/spark/spark-interpreter-0.9.0-│
preview2.jar 10.15.145.26 17317 "spark-fialkovskiy" :+ pid=8070                                                                                                                                           │
 INFO [2021-02-10 18:34:16,955] ({Exec Stream Pumper} ProcessLauncher.java[processLine]:188) - Interpreter launch command:  /usr/lib/spark/3.0.1/bin/spark-submit --class org.apache.zeppelin.interpreter.│
remote.RemoteInterpreterServer --driver-class-path ":/usr/lib/zeppelin/interpreter/spark/*::/usr/lib/zeppelin/interpreter/zeppelin-interpreter-shaded-0.9.0-preview2.jar:/usr/lib/zeppelin/interpreter/spa│
rk/spark-interpreter-0.9.0-preview2.jar:/etc/hadoop/" --driver-java-options " -Dfile.encoding=UTF-8 -Dlog4j.configuration='file:///etc/zeppelin/log4j.properties' -Dlog4j.configurationFile='file:///etc/z│
eppelin/log4j2.properties' -Dzeppelin.log.file='/usr/lib/zeppelin/logs/zeppelin-interpreter-spark-fialkovskiy-fialkovskiy--hadoop836713.log'" /usr/lib/zeppelin/interpreter/spark/spark-interpreter-0.9.0-│
preview2.jar 10.15.145.26 17317 "spark-fialkovskiy" :+ pid=8070                                                                                                                                           │
 INFO [2021-02-10 18:34:24,844] ({Thread-1034} ClusterManager.java[getIntpProcessStatus]:455) - interpreter thrift:17305 service is online!                                                  │
 INFO [2021-02-10 18:34:24,845] ({Thread-1034} ClusterManager.java[getIntpProcessStatus]:461) - interpreter thrift:17305 accessible!                                                         │
 INFO [2021-02-10 18:34:24,845] ({Thread-1034} ClusterInterpreterCheckThread.java[online]:62) - Found cluster interpreter:17305                                                              │
 INFO [2021-02-10 18:34:24,851] ({Thread-1034} ProcessLauncher.java[transition]:109) - Process state is transitioned to RUNNING                                                                           │
 INFO [2021-02-10 18:34:24,852] ({Thread-1034} ClusterInterpreterCheckThread.java[run]:81) - ClusterInterpreterCheckThread run() <<<                                                                      │
 INFO [2021-02-10 18:34:24,854] ({SchedulerFactory2} ClusterManager.java[getIntpProcessStatus]:455) - interpreter thrift 10.15.145.26:17305 service is online!                                            │
 INFO [2021-02-10 18:34:24,854] ({SchedulerFactory2} ClusterManager.java[getIntpProcessStatus]:461) - interpreter thrift 10.15.145.26:17305 accessible!                                                   │
 INFO [2021-02-10 18:34:24,861] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.SparkInterpreter                          │
 INFO [2021-02-10 18:34:24,960] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.SparkSqlInterpreter                       │
 INFO [2021-02-10 18:34:24,962] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.PySparkInterpreter                        │
 INFO [2021-02-10 18:34:24,966] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.IPySparkInterpreter                       │
 INFO [2021-02-10 18:34:24,970] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.SparkRInterpreter                         │
 INFO [2021-02-10 18:34:24,972] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.SparkIRInterpreter                        │
 INFO [2021-02-10 18:34:24,974] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.SparkShinyInterpreter                     │
 INFO [2021-02-10 18:34:24,975] ({SchedulerFactory2} RemoteInterpreter.java[lambda$internal_create$1]:166) - Create RemoteInterpreter org.apache.zeppelin.spark.KotlinSparkInterpreter                    │
 INFO [2021-02-10 18:34:25,050] ({SchedulerFactory2} RemoteInterpreter.java[lambda$open$0]:139) - Open RemoteInterpreter org.apache.zeppelin.spark.PySparkInterpreter                                     │
 INFO [2021-02-10 18:34:25,050] ({SchedulerFactory2} RemoteInterpreter.java[pushAngularObjectRegistryToRemote]:408) - Push local angular object registry from ZeppelinServer to remote interpreter group s│
park-fialkovskiy                                                                                                                                                                                          │
 INFO [2021-02-10 18:34:25,128] ({JobStatusPoller-paragraph_1612823207233_789036296} NotebookServer.java[onStatusChange]:1907) - Job paragraph_1612823207233_789036296 starts to RUNNING                  │
 WARN [2021-02-10 18:34:28,469] ({SchedulerFactory2} NotebookServer.java[onStatusChange]:1904) - Job paragraph_1612823207233_789036296 is finished, status: ERROR, exception: null, result: %text org.apac│
he.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter
       at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)                                                                                                          │
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:760)                                                                           │
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:668)                                                                           │
        at org.apache.zeppelin.scheduler.Job.run(Job.java:172)                                                                                                                                            │
        at org.apache.zeppelin.scheduler.AbstractScheduler.runJob(AbstractScheduler.java:130)                                                                                                             │
        at org.apache.zeppelin.scheduler.FIFOScheduler.lambda$runJobInScheduler$0(FIFOScheduler.java:39)                                                                                                  │
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)                                                                                                                │
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                                                                                                                │
        at java.lang.Thread.run(Thread.java:748)                                                                                                                                                          │
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter                                                      │
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:76)                                                                                                          │
        at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:355)                                                                                    │
        at org.apache.zeppelin.interpreter.Interpreter.getInterpreterInTheSameSessionByClassName(Interpreter.java:366)                                                                                    │
        at org.apache.zeppelin.spark.PySparkInterpreter.open(PySparkInterpreter.java:89)                                                                                                                  │
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)                                                                                                          │
        ... 8 more                                                                                                                                                                                        │
Caused by: org.apache.zeppelin.interpreter.InterpreterException: Fail to open SparkInterpreter                                                                                                            │
        at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:122)                                                                                                                     │
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)                                                                                                          │
        ... 12 more                                                                                                                                                                                       │
Caused by: java.lang.reflect.InvocationTargetException                                                                                                                                                    │
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                                                                                    │
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                                                                                  │
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                                                                                          │
        at java.lang.reflect.Method.invoke(Method.java:498)                                                                                                                                               │
        at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.spark2CreateContext(BaseSparkScalaInterpreter.scala:301)                                                                                   │
        at org.apache.zeppelin.spark.BaseSparkScalaInterpreter.createSparkContext(BaseSparkScalaInterpreter.scala:230)                                                                                    │
        at org.apache.zeppelin.spark.SparkScala212Interpreter.open(SparkScala212Interpreter.scala:90)                                                                                                     │
        at org.apache.zeppelin.spark.SparkInterpreter.open(SparkInterpreter.java:106)                                                                                                                     │
        ... 13 more                                                                                                                                                                                       │
Caused by: org.apache.spark.SparkException: Detected yarn cluster mode, but isn't running on a cluster. Deployment to YARN is not supported directly by SparkContext. Please use spark-submit.            │
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:402)                                                                                                                                   │
        at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2574)                                                                                                                            │
        at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:934)                                                                                                       │
        at scala.Option.getOrElse(Option.scala:189)                                                                                                                                                       │
        at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:928)                                                                                                                  │
        ... 21 more