[ZEPPELIN-4310] Shell interpreter fails to execute on new installation of Zeppelin

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.1
• Component/s: build, Interpreters
• Assignee: Unassigned
• Reporter: Tim Ojo
• Created: 21/Aug/19 20:15
• Updated: 22/Aug/19 01:16

Description

I just downloaded and installed the netinst binary package of zeppelin 0.8.1 and installed the shell interpreter with the command:

./bin/install-interpreter.sh --name shell

 

If I open up a new note and try to run a shell command

%sh

ls -lh

 

I get the following error.

INFO [2019-08-21 13:27:05,832] ({qtp1582797472-104} RemoteInterpreter.java[call]:168) - Create RemoteInterpreter org.apache.zep
pelin.shell.ShellInterpreter
 INFO [2019-08-21 13:27:05,928] ({qtp1582797472-104} NotebookServer.java[completion]:1413) - Fail to get completion
java.lang.RuntimeException: org.apache.thrift.TApplicationException: Internal error processing createInterpreter
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:139)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:165)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:132)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.completion(RemoteInterpreter.java:349)
        at org.apache.zeppelin.notebook.Paragraph.completion(Paragraph.java:270)
        at org.apache.zeppelin.notebook.Note.completion(Note.java:729)
        at org.apache.zeppelin.socket.NotebookServer.completion(NotebookServer.java:1411)
        at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:304)
        at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:189)
        at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:66)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:158)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:162)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:459)
        at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:182)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:105)
        at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:142)
        at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:85)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:220)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:219)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:244)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:559)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:390)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
        at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
        at java.lang.Thread.run(Thread.java:745)

 

In the shell interpreter logs it translates to a ClassCastException

ERROR [2019-08-21 13:27:05,926] ({pool-1-thread-5} ProcessFunction.java[process]:41) - Internal error processing createInterpret
er
org.apache.thrift.TException: java.lang.ClassNotFoundException: org.apache.zeppelin.shell.ShellInterpreter
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.createInterpreter(RemoteInterpreterServer.java:320)
        at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(RemoteInterpret
erService.java:1799)
        at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$createInterpreter.getResult(RemoteInterpret
erService.java:1784)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.zeppelin.shell.ShellInterpreter
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:260)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.createInterpreter(RemoteInterpreterServer.java:302)
        ... 8 more

 

After a little googling I found out that the issue is that zeppelin looks for a specific path when attempting to instantiate interpreters. However, when the shell interpreter is installed using the `install-interpreter.sh` script, it installs it to the `interpreter/shell` folder but the plugin system looks for the classes in the `interpreter/sh` folder.

A simple workaround for this issue is to rename the shell folder (`mv interpreter/shell interpreter/sh`) and restart the zeppelin daemon. But it would be good to fix the plugin system to look for the shell classes in the `interpreter/shell` directory instead of having manually rename the folder.