[ZEPPELIN-5643] deadlock in getNote

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.0
• Component/s: zeppelin-server

Environment

0.10.0

• Flags: Important
• Assignee: Unassigned
• Reporter: zhaoming.chen
• Created: 27/Jan/22 08:47
• Updated: 28/Jan/22 06:06

Description

I can login the zeppelin, but can't enter note. the stack info of zeppelin-server process like this:

Found one Java-level deadlock:
=============================
"qtp1593180232-1490":
  waiting to lock monitor 0x00007fcba40055b8 (object 0x0000000681523a68, a java.util.HashMap),
  which is held by "qtp1593180232-819"
"qtp1593180232-819":
  waiting to lock monitor 0x00007fcba4005668 (object 0x00000006a2988898, a org.apache.zeppelin.socket.NotebookSocket),
  which is held by "qtp1593180232-795"
"qtp1593180232-795":
  waiting to lock monitor 0x00007fcba40055b8 (object 0x0000000681523a68, a java.util.HashMap),
  which is held by "qtp1593180232-819"

Java stack information for the threads listed above:

"qtp1593180232-1490" #1490 prio=5 os_prio=0 tid=0x00007fcadc013000 nid=0x45f5 waiting for monitor entry [0x00007fca956c9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:171)
    - waiting to lock <0x0000000681523a68> (a java.util.HashMap)
    at org.apache.zeppelin.socket.NotebookServer$9.onSuccess(NotebookServer.java:888)
    at org.apache.zeppelin.socket.NotebookServer$9.onSuccess(NotebookServer.java:879)
    at org.apache.zeppelin.service.NotebookService.getHomeNote(NotebookService.java:114)
    at org.apache.zeppelin.socket.NotebookServer.getHomeNote(NotebookServer.java:878)
    at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:308)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)

    at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.lang.Thread.run(Thread.java:748)

"qtp1593180232-819" #819 prio=5 os_prio=0 tid=0x00007fcb3c003800 nid=0x3967 waiting for monitor entry [0x00007fcb94fd0000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
    - waiting to lock <0x00000006a2988898> (a org.apache.zeppelin.socket.NotebookSocket)
    at org.apache.zeppelin.socket.ConnectionManager.broadcast(ConnectionManager.java:242)
    at org.apache.zeppelin.socket.ConnectionManager.checkCollaborativeStatus(ConnectionManager.java:209)
    at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromNote(ConnectionManager.java:185)
    - locked <0x0000000681523a68> (a java.util.HashMap)
    at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:173)
    - locked <0x0000000681523a68> (a java.util.HashMap))
    at org.apache.zeppelin.socket.ConnectionManager.addNoteConnection(ConnectionManager.java:107)
    - locked <0x0000000681523a68> (a java.util.HashMap)
    at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:822)
    at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
    at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
    at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
    at org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
    at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)

    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
    at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.lang.Thread.run(Thread.java:748)

"qtp1593180232-795" #795 prio=5 os_prio=0 tid=0x00007fcae00ef000 nid=0x3944 waiting for monitor entry [0x00007fcb958d2000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.zeppelin.socket.ConnectionManager.removeConnectionFromAllNote(ConnectionManager.java:171)
    - waiting to lock <0x0000000681523a68> (a java.util.HashMap)
    at org.apache.zeppelin.socket.NotebookServer.onClose(NotebookServer.java:476)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketClose(NotebookSocket.java:47)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onClose(JettyListenerEventDriver.java:119)
    at org.eclipse.jetty.websocket.common.WebSocketSession.callApplicationOnClose(WebSocketSession.java:393)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:225)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection$Flusher.onCompleteFailure(AbstractWebSocketConnection.java:100)
    at org.eclipse.jetty.util.IteratingCallback.failed(IteratingCallback.java:402)
    at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:302)
    at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
    at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
    at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextOutgoingFrame(AbstractExtension.java:155)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextOutgoingFrame(PerMessageDeflateExtension.java:123)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.access$1100(CompressExtension.java:44)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.compress(CompressExtension.java:591)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.deflate(CompressExtension.java:488)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension$Flusher.process(CompressExtension.java:468)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.outgoingFrame(CompressExtension.java:244)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack$Flusher.process(ExtensionStack.java:400)
    at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
    at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.outgoingFrame(ExtensionStack.java:277)
    at org.eclipse.jetty.websocket.common.WebSocketSession.outgoingFrame(WebSocketSession.java:359)
    at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.uncheckedSendFrame(WebSocketRemoteEndpoint.java:306)
    at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendAsyncFrame(WebSocketRemoteEndpoint.java:240)
    at org.eclipse.jetty.websocket.common.WebSocketRemoteEndpoint.sendStringByFuture(WebSocketRemoteEndpoint.java:403)
    at org.apache.zeppelin.socket.NotebookSocket.send(NotebookSocket.java:70)
    - locked <0x00000006a2988898> (a org.apache.zeppelin.socket.NotebookSocket)
    at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:823)
    at org.apache.zeppelin.socket.NotebookServer$7.onSuccess(NotebookServer.java:819)
    at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:141)
    at org.apache.zeppelin.service.NotebookService.getNote(NotebookService.java:121)
    at org.apache.zeppelin.socket.NotebookServer.getNote(NotebookServer.java:818)
    at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:311)
    at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:58)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:231)
    at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
    at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:179)
    at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:150)
    at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326)
    at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:147)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMessageDeflateExtension.java:112)
    at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtension.java:168)
    at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessageDeflateExtension.java:92)
    at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202)
    at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225)
    at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:459)
    at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.lang.Thread.run(Thread.java:748)