Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2997

Add configuration for disabling the feature of "auto-restart interpreter on cron execution"

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Add configuration for disabling the feature of "auto-restart interpreter on cron execution".

      There is a risk of deadlock when a number of Notebooks which same interpreters are bound are scheduled to run at the same time and the "auto-restart interpreter on cron execution" checkbox is checked on some of them.

      Configuration for disabling the feature of "auto-restart interpreter on cron execution" should be added on Zeppelin.

      Here is the deadlock information of the jstack command when the deadlock occurs in the ZeppelinServer process:

      Found one Java-level deadlock:
      =============================
      "pool-2-thread-63":
        waiting to lock monitor 0x00007fd2fc0073e8 (object 0x00000000c04f9c50, a java.util.concurrent.ConcurrentHashMap),
        which is held by "DefaultQuartzScheduler_Worker-2"
      "DefaultQuartzScheduler_Worker-2":
        waiting to lock monitor 0x00007fd28c0036c8 (object 0x00000000c16f4738, a org.apache.zeppelin.notebook.Note),
        which is held by "DefaultQuartzScheduler_Worker-4"
      "DefaultQuartzScheduler_Worker-4":
        waiting to lock monitor 0x00007fd2fc0073e8 (object 0x00000000c04f9c50, a java.util.concurrent.ConcurrentHashMap),
        which is held by "DefaultQuartzScheduler_Worker-2"
      
      Java stack information for the threads listed above:
      ===================================================
      "pool-2-thread-63":
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:981)
              - waiting to lock <0x00000000c04f9c50> (a java.util.concurrent.ConcurrentHashMap)
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.getInterpreterSettings(InterpreterSettingManager.java:450)
              at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:382)
              at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:255)
              at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:361)
              at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
              at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      "DefaultQuartzScheduler_Worker-2":
              at org.apache.zeppelin.notebook.Note.stopDelayedPersistTimer(Note.java:789)
              - waiting to lock <0x00000000c16f4738> (a org.apache.zeppelin.notebook.Note)
              at org.apache.zeppelin.notebook.Note.persist(Note.java:725)
              at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:2070)
              at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:149)
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.stopJobAllInterpreter(InterpreterSettingManager.java:966)
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:942)
              - locked <0x00000000c04f9c50> (a java.util.concurrent.ConcurrentHashMap)
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.restart(InterpreterSettingManager.java:956)
              at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:907)
              at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
              at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
              - locked <0x00000000c0ef6190> (a java.lang.Object)
      "DefaultQuartzScheduler_Worker-4":
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.get(InterpreterSettingManager.java:981)
              - waiting to lock <0x00000000c04f9c50> (a java.util.concurrent.ConcurrentHashMap)
              at org.apache.zeppelin.interpreter.InterpreterSettingManager.getInterpreterSettings(InterpreterSettingManager.java:450)
              at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:382)
              at org.apache.zeppelin.notebook.Paragraph.isValidInterpreter(Paragraph.java:701)
              at org.apache.zeppelin.notebook.Paragraph.getMagic(Paragraph.java:690)
              at org.apache.zeppelin.notebook.Paragraph.isBlankParagraph(Paragraph.java:354)
              at org.apache.zeppelin.notebook.Note.run(Note.java:609)
              at org.apache.zeppelin.notebook.Note.runAll(Note.java:596)
              at org.apache.zeppelin.notebook.Note.runAll(Note.java:587)
              - locked <0x00000000c16f4738> (a org.apache.zeppelin.notebook.Note)
              at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885)
              at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
              at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
              - locked <0x00000000c0f4b800> (a java.lang.Object)
      
      Found 1 deadlock.
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kjmrknsn Keiji Yoshida
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: