[ZEPPELIN-5495] The interpreter cannot be used normally in Zeppelin cluster mode

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0
• Component/s: zeppelin-interpreter
• Assignee: Unassigned
• Reporter: xietianci
• Created: 24/Aug/21 03:41
• Updated: 24/Aug/21 03:44

Description

I tried Zeppelin's cluster mode according to the documentation。https://docs.google.com/document/d/1a8QLSyR3M5AhlG1GIYuDTj6bwazeuVDKCRRBm-Qa3Bw/edit#heading=h.s41ckl271z8s

The  feature of cluster mode is very cool, and I successfully started the cluster. But there was a problem when using the interpreter.The error is as follows.

 

INFO [2021-08-24 10:28:38,005] ({main} PluginManager.java[loadInterpreterLauncher]:154) - Loading Interpreter Launcher Plugin: org.apache.zeppelin.interpreter.launcher.ClusterInterpreterLauncher  INFO [2021-08-24 10:28:38,015] ({Thread-7} ClusterManagerServer.java[run]:161) - RaftServer run() >>>  INFO [2021-08-24 10:28:38,030] ({Thread-9} ClusterManager.java[run]:221) - RaftClientThread run() >>>  INFO [2021-08-24 10:28:38,031] ({main} ZeppelinServer.java[main]:249) - Starting zeppelin server  INFO [2021-08-24 10:28:38,033] ({main} Server.java[doStart]:360) - jetty-9.4.31.v20200723; built: 2020-07-23T17:57:36.812Z; git: 450ba27947e13e66baa8cd1ce7e85a4461cacc1d; jvm 1.8.0_262-b10 ERROR [2021-08-24 10:28:38,039] ({Thread-8} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!  WARN [2021-08-24 10:28:38,039] ({Thread-8} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.  WARN [2021-08-24 10:28:38,052] ({Thread-9} NettyMessagingService.java[loadKeyStores]:270) - Disabling TLS for intra-cluster messaging; Could not load cluster key store: ../config/atomix.jks (没有那个文件或目录)  WARN [2021-08-24 10:28:38,052] ({Thread-7} NettyMessagingService.java[loadKeyStores]:270) - Disabling TLS for intra-cluster messaging; Could not load cluster key store: ../config/atomix.jks (没有那个文件或目录)  INFO [2021-08-24 10:28:38,132] ({main} StandardDescriptorProcessor.java[visitServlet]:276) - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet  INFO [2021-08-24 10:28:38,142] ({main} DefaultSessionIdManager.java[doStart]:334) - DefaultSessionIdManager workerName=node0  INFO [2021-08-24 10:28:38,142] ({main} DefaultSessionIdManager.java[doStart]:339) - No SessionScavenger set, using defaults  INFO [2021-08-24 10:28:38,144] ({main} HouseKeeper.java[startScavenging]:140) - node0 Scavenging every 600000ms  INFO [2021-08-24 10:28:39,238] ({main} ContextHandler.java[doStart]:860) - Started o.e.j.w.WebAppContext@18f8cd79 {zeppelin-web,/,jar:file:///home/Zeppelin/zeppelin-0.9.0-bin-all/zeppelin-web-0.9.0.war!/,AVAILABLE} {/home/Zeppelin/zeppelin-0.9.0-bin-all/zeppelin-web-0.9.0.war} INFO [2021-08-24 10:28:39,253] ({main} StandardDescriptorProcessor.java[visitServlet]:276) - NO JSP Support for /next, did not find org.eclipse.jetty.jsp.JettyJspServlet  INFO [2021-08-24 10:28:39,444] ({main} ContextHandler.java[doStart]:860) - Started o.e.j.w.WebAppContext@61710c6 {zeppelin-web-angular,/next,jar:[file:///home/Zeppelin/zeppelin-0.9.0-bin-all/zeppelin-web-angular-0.9.0.war!/,AVAILABLE]} {/home/Zeppelin/zeppelin-0.9.0-bin-all/zeppelin-web-angular-0.9.0.war}  INFO [2021-08-24 10:28:39,490] ({main} AbstractConnector.java[doStart]:331) - Started ServerConnector@61df66b6{HTTP/1.1, (http/1.1)} {0.0.0.0:8093}  INFO [2021-08-24 10:28:39,491] ({main} Server.java[doStart]:400) - Started @4339ms  INFO [2021-08-24 10:28:40,308] ({netty-messaging-event-epoll-server-0} NettyMessagingService.java[lambda$startAcceptingConnections$24]:605) - /10.128.143.162 accepting incoming connections on port 6000  INFO [2021-08-24 10:28:40,308] ({netty-messaging-event-epoll-server-0} NettyMessagingService.java[lambda$startAcceptingConnections$24]:605) - /10.128.143.162 accepting incoming connections on port 27887  INFO [2021-08-24 10:28:40,309] ({netty-messaging-event-epoll-server-0} NettyMessagingService.java[lambda$start$0]:225) - Started  INFO [2021-08-24 10:28:40,309] ({netty-messaging-event-epoll-server-0} NettyMessagingService.java[lambda$start$0]:225) - Started  INFO [2021-08-24 10:28:40,412] ({qtp1115201599-72} NotebookServer.java[onOpen]:247) - New connection from 10.128.136.12:57898 ERROR [2021-08-24 10:28:41,040] ({Thread-8} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!  WARN [2021-08-24 10:28:41,040] ({Thread-8} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.  WARN [2021-08-24 10:28:41,046] ({Thread-10} ClusterManager.java[run]:266) - Raft incomplete initialization! retry[30] ERROR [2021-08-24 10:28:42,551] ({Thread-7} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!  WARN [2021-08-24 10:28:42,551] ({Thread-7} ClusterManager.java[putClusterMeta]:369) - putClusterMeta failure, Cache metadata to queue.  INFO [2021-08-24 10:28:42,551] ({Thread-7} ClusterManagerServer.java[run]:224) - RaftServer run() <<<  INFO [2021-08-24 10:28:42,552] ({raft-server-10.128.143.162:6000} DelegatingLogger.java[info]:170) - RaftServer{10.128.143.162:6000} - Transitioning to FOLLOWER  INFO [2021-08-24 10:28:43,816] ({raft-server-10.128.143.162:6000} DelegatingLogger.java[info]:170) - RaftServer{10.128.143.162:6000} - Transitioning to CANDIDATE  INFO [2021-08-24 10:28:43,818] ({raft-server-10.128.143.162:6000} DelegatingLogger.java[info]:165) - RaftServer{10.128.143.162:6000} {role=CANDIDATE} Starting election  INFO [2021-08-24 10:28:43,846] ({raft-server-10.128.143.162:6000} DelegatingLogger.java[info]:170) - RaftServer{10.128.143.162:6000} - Transitioning to LEADER  INFO [2021-08-24 10:28:43,852] ({raft-server-10.128.143.162:6000} DelegatingLogger.java[info]:170) - RaftServer{10.128.143.162:6000} - Found leader 10.128.143.162:6000 ERROR [2021-08-24 10:28:44,041] ({Thread-8} ClusterManager.java[putClusterMeta]:341) - Raft incomplete initialization!

 

 

WARN [2021-08-24 10:31:20,260] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:21,263] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:22,267] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:23,277] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  INFO [2021-08-24 10:31:23,302] ({Thread-143} ClusterInterpreterCheckThread.java[run]:51) - ClusterInterpreterCheckThread run() >>>  WARN [2021-08-24 10:31:23,303] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  INFO [2021-08-24 10:31:23,311] ({netty-messaging-event-epoll-client-13} ProcessLauncher.java[transition]:109) - Process state is transitioned to LAUNCHED  INFO [2021-08-24 10:31:23,311] ({netty-messaging-event-epoll-client-13} ProcessLauncher.java[launch]:96) - Process is launched: [/home/Zeppelin/zeppelin-0.9.0-bin-all/bin/interpreter.sh, -d, /home/Zeppelin/zeppelin-0.9.0-bin-all/interpreter/sh, -c, 10.130.11.97, -p, 30435, -r, :, -i, sh-shared_process, -l, /home/Zeppelin/zeppelin-0.9.0-bin-all/local-repo/sh, -g, sh]  WARN [2021-08-24 10:31:24,279] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:24,305] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:25,282] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:25,306] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:26,284] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:26,307] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process  WARN [2021-08-24 10:31:27,286] ({raft-server-10.128.143.162:6000-state} ClusterMeta.java[get]:117) - can not find key : sh-shared_process