[ZEPPELIN-3698] Zeppelin stops working after a few days of uptime with RemoteEndpoint unavailable, outgoing connection not open

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.1, 0.9.0
• Assignee: George Klimov
• Reporter: Clemens Valiente
• Created: 09/Aug/18 13:22
• Updated: 23/Jan/19 02:47
• Resolved: 12/Sep/18 11:38

Description

I am running a zeppelin 0.9.0-snapshot instance and it works fine most of the time, but after running it for a while we eventually encounter issues. The notebooks suddenly show up empty (are not loading) and the logfile (apache-zeppelin-dev.log) shows the following error: INFO [2018-07-13 09:57:38,484] ({qtp2091156596-404} NotebookServer.java[sendNote]:865) - New operation from 172.21.95.21 : 62462 : johndoe : GET_NOTE : 2DJK3XG1R ERROR [2018-07-13 09:57:38,484] ({qtp2091156596-404} NotebookServer.java[onMessage]:395) - Can't handle message: {"op":"GET_NOTE","data":

{"id":"2DJK3XG1R"}

,"principal":"johndoe","ticket":"52ceedde-5654-49b7-88c1-bcd95ec9ae8c","roles":"[]"} org.eclipse.jetty.websocket.api.WebSocketException: RemoteEndpoint unavailable, outgoing connection not open at org.eclipse.jetty.websocket.common.WebSocketSession.getRemote(WebSocketSession.java:252) at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70) at org.apache.zeppelin.socket.NotebookServer.broadcast(NotebookServer.java:553) at org.apache.zeppelin.socket.NotebookServer.checkCollaborativeStatus(NotebookServer.java:496) at org.apache.zeppelin.socket.NotebookServer.removeConnectionFromNote(NotebookServer.java:457) at org.apache.zeppelin.socket.NotebookServer.removeConnectionFromAllNote(NotebookServer.java:471) at org.apache.zeppelin.socket.NotebookServer.addConnectionToNote(NotebookServer.java:437) at org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:884) at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:249) at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:748) after restarting the server with zeppelin-daemon.sh restart, everything works again (for a while). this happens for all users on different systems, after logging in/logging out, so it doesn't look like a client related problem. The little websocket indicator next to the username in the webUI shows up green, so I don't know which websocket this error is referring to. I didn't find this error in any tasks or reported issue. Did anyone else encounter this problem? Any hints on how I can further debug this?

Issue Links

• links to: GitHub Pull Request #3167