[ZEPPELIN-3193] Zeppelin 0.7.3 Job run deadlock

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.7.3
• Component/s: Interpreters
• Assignee: Unassigned
• Reporter: ganguowei
• Created: 29/Jan/18 09:03
• Updated: 18/Sep/23 02:06

Description

dump :

Found one Java-level deadlock:
=============================
"qtp2008362258-12861":
waiting to lock monitor 0x00007fc44400da28 (object 0x00000000c036c8e0, a java.util.concurrent.ConcurrentHashMap),
which is held by "DefaultQuartzScheduler_Worker-5"
"DefaultQuartzScheduler_Worker-5":
waiting to lock monitor 0x00007fc458028438 (object 0x00000000c0bd0548, a org.apache.zeppelin.notebook.Note),
which is held by "DefaultQuartzScheduler_Worker-2"
"DefaultQuartzScheduler_Worker-2":
waiting to lock monitor 0x00007fc44400da28 (object 0x00000000c036c8e0, a java.util.concurrent.ConcurrentHashMap),
which is held by "DefaultQuartzScheduler_Worker-5"

Java stack information for the threads listed above:
===================================================
"qtp2008362258-12861":
at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:981)

• waiting to lock <0x00000000c036c8e0> (a java.util.concurrent.ConcurrentHashMap)
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.getInterpreterSettings(InterpreterSettingManager.java:450)
  at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:852)
  at org.apache.zeppelin.socket.NotebookServer.unicastNoteJobInfo(NotebookServer.java:532)
  at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:334)
  at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
  at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
  at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
  at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
  at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
  at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
  at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
  at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
  at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
  at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
  at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
  at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
  at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
  at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
  at java.lang.Thread.run(Thread.java:745)
  "DefaultQuartzScheduler_Worker-5":
  at org.apache.zeppelin.notebook.Note.stopDelayedPersistTimer(Note.java:789)
• waiting to lock <0x00000000c0bd0548> (a org.apache.zeppelin.notebook.Note)
  at org.apache.zeppelin.notebook.Note.persist(Note.java:725)
  at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:2070)
  at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:149)
  at org.apache.zeppelin.notebook.Paragraph.jobAbort(Paragraph.java:473)
  at org.apache.zeppelin.scheduler.Job.abort(Job.java:240)
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.stopJobAllInterpreter(InterpreterSettingManager.java:965)
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:942)
• locked <0x00000000c036c8e0> (a java.util.concurrent.ConcurrentHashMap)
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:956)
  at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:907)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
• locked <0x00000000c07232c8> (a java.lang.Object)
  "DefaultQuartzScheduler_Worker-2":
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:981)
• waiting to lock <0x00000000c036c8e0> (a java.util.concurrent.ConcurrentHashMap)
  at org.apache.zeppelin.interpreter.InterpreterSettingManager.getInterpreterSettings(InterpreterSettingManager.java:450)
  at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:382)
  at org.apache.zeppelin.notebook.Note.run(Note.java:616)
  at org.apache.zeppelin.notebook.Note.runAll(Note.java:596)
  at org.apache.zeppelin.notebook.Note.runAll(Note.java:587)
• locked <0x00000000c0bd0548> (a org.apache.zeppelin.notebook.Note)
  at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
• locked <0x00000000c0722830> (a java.lang.Object)