[ZEPPELIN-3499] Deadlock between Interpreter restart and JobProgressPoller

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.7.3
• Component/s: zeppelin-server
• Assignee: Unassigned
• Reporter: Prabhu Joseph
• Created: 24/May/18 11:12
• Updated: 24/May/18 11:12

Description

Zeppelin Server hangs due to a deadlock

"qtp1146147158-107615":
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:972)
        - waiting to lock <0x00000000c0611a10> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.getInterpreterSettings(InterpreterSettingManager.java:441)
        at org.apache.zeppelin.socket.NotebookServer.sendAllAngularObjects(NotebookServer.java:2133)
        at org.apache.zeppelin.socket.NotebookServer.sendNote(NotebookServer.java:736)
        at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:227)
        at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
        at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
        at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
        at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
        at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
        at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
        at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
        at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
        at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
"DefaultQuartzScheduler_Worker-10":
        at org.apache.zeppelin.interpreter.InterpreterGroup.getId(InterpreterGroup.java:98)
        - waiting to lock <0x00000000cbdc6898> (a org.apache.zeppelin.interpreter.InterpreterGroup)
        at org.apache.zeppelin.notebook.Note.snapshotAngularObjectRegistry(Note.java:682)
        at org.apache.zeppelin.notebook.Note.persist(Note.java:727)
        at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:2073)
        at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:149)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.stopJobAllInterpreter(InterpreterSettingManager.java:957)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:933)
        - locked <0x00000000c0611a10> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:947)
        at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:907)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
        - locked <0x00000000c0596ae8> (a java.lang.Object)
"Thread-102262":
        at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:972)
        - waiting to lock <0x00000000c0611a10> (a java.util.concurrent.ConcurrentHashMap)
        at org.apache.zeppelin.interpreter.InterpreterFactory.createRemoteRepl(InterpreterFactory.java:304)
        at org.apache.zeppelin.interpreter.InterpreterFactory.createInterpretersForNote(InterpreterFactory.java:202)
        at org.apache.zeppelin.interpreter.InterpreterFactory.createOrGetInterpreterList(InterpreterFactory.java:333)
        - locked <0x00000000cbdc6898> (a org.apache.zeppelin.interpreter.InterpreterGroup)
        at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:372)
        at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:424)
        at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:256)
        at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:331)
        at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:51)

Found 1 deadlock.