Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2321

zeppelin hungup when executing cronjob

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: None
    • Component/s: Interpreters
    • Labels:
    • Environment:

      hadoop 2.7.3
      spark 2.1.0
      jdk8

    • Docs Text:
      Hide
      2017-03-27 16:27:06
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode):

      "Attach Listener" #96 daemon prio=9 os_prio=0 tid=0x00007f1a80001000 nid=0x14576 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "qtp759156157-95" #95 prio=5 os_prio=0 tid=0x00007f19c0002000 nid=0x14543 waiting on condition [0x00007f1add2fe000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c00a2de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-94" #94 prio=5 os_prio=0 tid=0x00007f19c0001000 nid=0x14542 waiting for monitor entry [0x00007f1add0f9000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317)
      at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:241)
      at org.apache.zeppelin.notebook.Paragraph.getCurrentRepl(Paragraph.java:245)
      at org.apache.zeppelin.helium.Helium.suggestApp(Helium.java:249)
      at org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:84)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
      at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:205)
      at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:102)
      at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
      at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)
      at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
      - locked <0x00000000ee0be198> (a org.apache.cxf.phase.PhaseInterceptorChain)
      at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
      at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
      at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
      at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
      at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:575)
      at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
      at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
      at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
      at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
      at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
      at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
      at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
      at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
      at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
      at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
      at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
      at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
      at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72)
      at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
      at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
      at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      at org.eclipse.jetty.server.Server.handle(Server.java:499)
      at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "pool-2-thread-4" #93 prio=5 os_prio=0 tid=0x00007f1990001000 nid=0x14513 waiting on condition [0x00007f1adc858000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "pool-2-thread-3" #91 prio=5 os_prio=0 tid=0x00007f1998001000 nid=0x14511 waiting on condition [0x00007f1ad7ffe000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "pool-4-thread-1" #85 prio=5 os_prio=0 tid=0x00007f19a0001800 nid=0x1449f waiting on condition [0x00007f1adca5a000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000eb62fe18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
      at org.apache.zeppelin.interpreter.remote.AppendOutputRunner.run(AppendOutputRunner.java:66)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "Thread-41" #77 prio=5 os_prio=0 tid=0x00007f19b802e800 nid=0x1449c waiting for monitor entry [0x00007f1adcb5b000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.get(InterpreterFactory.java:891)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.socket.NotebookServer.onMetaInfosReceived(NotebookServer.java:2181)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:241)

      "Exec Stream Pumper" #84 daemon prio=5 os_prio=0 tid=0x00007f19b4003000 nid=0x14433 runnable [0x00007f1ad7bfa000]
         java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:255)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      - locked <0x00000000c126c760> (a java.lang.UNIXProcess$ProcessPipeInputStream)
      at java.io.FilterInputStream.read(FilterInputStream.java:107)
      at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
      at java.lang.Thread.run(Thread.java:745)

      "Exec Stream Pumper" #83 daemon prio=5 os_prio=0 tid=0x00007f19b4002000 nid=0x14432 runnable [0x00007f1ad7cfb000]
         java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:255)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      - locked <0x00000000c121a900> (a java.lang.UNIXProcess$ProcessPipeInputStream)
      at java.io.FilterInputStream.read(FilterInputStream.java:107)
      at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107)
      at java.lang.Thread.run(Thread.java:745)

      "Exec Default Executor" #82 prio=5 os_prio=0 tid=0x00007f19b8011000 nid=0x14430 in Object.wait() [0x00007f1ad7dfc000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:502)
      at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396)
      - locked <0x00000000c1265e90> (a java.lang.UNIXProcess)
      at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364)
      at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48)
      at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200)
      at java.lang.Thread.run(Thread.java:745)

      "Thread-43" #81 prio=5 os_prio=0 tid=0x00007f19b8003800 nid=0x1442f waiting for monitor entry [0x00007f1ad7efd000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317)
      at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:241)
      at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:309)
      at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:51)

      "pool-2-thread-2" #79 prio=5 os_prio=0 tid=0x00007f19c4001800 nid=0x1442d waiting for monitor entry [0x00007f1adc483000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.notebook.Note.snapshotAngularObjectRegistry(Note.java:624)
      at org.apache.zeppelin.notebook.Note.persist(Note.java:677)
      at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:2032)
      at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:149)
      at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:349)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "pool-2-thread-1" #78 prio=5 os_prio=0 tid=0x00007f19c8021800 nid=0x1442c waiting on condition [0x00007f1adc585000]
         java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-76" #76 prio=5 os_prio=0 tid=0x00007f19cc001000 nid=0x1442b waiting on condition [0x00007f1adcefa000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c00a2de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-75" #75 prio=5 os_prio=0 tid=0x00007f19c800a800 nid=0x1442a waiting for monitor entry [0x00007f1adcffa000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317)
      at org.apache.zeppelin.notebook.Paragraph.isValidInterpreter(Paragraph.java:673)
      at org.apache.zeppelin.notebook.Paragraph.getMagic(Paragraph.java:662)
      at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1607)
      at org.apache.zeppelin.socket.NotebookServer.runParagraph(NotebookServer.java:1600)
      at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:263)
      at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
      at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
      at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
      at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
      at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
      at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
      at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
      at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
      at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
      at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
      at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
      at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
      at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "Thread-37" #73 daemon prio=5 os_prio=0 tid=0x00007f19dc013800 nid=0x14428 waiting on condition [0x00007f1add1fd000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c124c050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "Scheduler-1566502717" #71 prio=5 os_prio=0 tid=0x00007f1a1c002000 nid=0x14425 waiting on condition [0x00007f1add500000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c1266320> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "Timer-0" #67 daemon prio=5 os_prio=0 tid=0x00007f1b2ae81800 nid=0x14420 in Object.wait() [0x00007f1ade4c7000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.util.TimerThread.mainLoop(Timer.java:552)
      - locked <0x00000000c0ad9a68> (a java.util.TaskQueue)
      at java.util.TimerThread.run(Timer.java:505)

      "DefaultQuartzScheduler_QuartzSchedulerThread" #66 prio=5 os_prio=0 tid=0x00007f1b2ae7e800 nid=0x1441f in Object.wait() [0x00007f1ade5c8000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410)
      - locked <0x00000000c0b64ac0> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-10" #65 prio=5 os_prio=0 tid=0x00007f1b28125000 nid=0x1441e in Object.wait() [0x00007f1ade6c9000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b72278> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-9" #64 prio=5 os_prio=0 tid=0x00007f1b28123000 nid=0x1441d in Object.wait() [0x00007f1ade7ca000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b7db50> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-8" #63 prio=5 os_prio=0 tid=0x00007f1b28121000 nid=0x1441c in Object.wait() [0x00007f1ade9cc000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b74000> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-7" #62 prio=5 os_prio=0 tid=0x00007f1b2813c000 nid=0x1441b in Object.wait() [0x00007f1ad7361000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b73c90> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-6" #61 prio=5 os_prio=0 tid=0x00007f1b2813a800 nid=0x1441a in Object.wait() [0x00007f1adc2b1000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b73920> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-5" #60 prio=5 os_prio=0 tid=0x00007f1b28139800 nid=0x14419 in Object.wait() [0x00007f1ade8cb000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
      - locked <0x00000000c0b735b0> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-4" #59 prio=5 os_prio=0 tid=0x00007f1b28138000 nid=0x14418 waiting for monitor entry [0x00007f1adc1b0000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      - locked <0x00000000c0b73240> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-3" #58 prio=5 os_prio=0 tid=0x00007f1b28137000 nid=0x14417 waiting for monitor entry [0x00007f1ad715f000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      - locked <0x00000000c0b72ed0> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-2" #57 prio=5 os_prio=0 tid=0x00007f1b289e0800 nid=0x14416 waiting for monitor entry [0x00007f1ad705e000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932)
      - waiting to lock <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199)
      at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      - locked <0x00000000c0b72b60> (a java.lang.Object)

      "DefaultQuartzScheduler_Worker-1" #56 prio=5 os_prio=0 tid=0x00007f1b289e1800 nid=0x14415 runnable [0x00007f1ad725f000]
         java.lang.Thread.State: RUNNABLE
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      at java.net.SocketInputStream.read(SocketInputStream.java:170)
      at java.net.SocketInputStream.read(SocketInputStream.java:141)
      at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
      at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      - locked <0x00000000eb696aa8> (a java.io.BufferedInputStream)
      at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
      at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
      at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
      at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
      at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
      at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_shutdown(RemoteInterpreterService.java:385)
      at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.shutdown(RemoteInterpreterService.java:373)
      at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.dereference(RemoteInterpreterProcess.java:128)
      - locked <0x00000000c12567a8> (a java.util.concurrent.atomic.AtomicInteger)
      at org.apache.zeppelin.interpreter.InterpreterGroup.close(InterpreterGroup.java:155)
      at org.apache.zeppelin.interpreter.InterpreterSetting.closeAndRemoveInterpreterGroupByNoteId(InterpreterSetting.java:238)
      at org.apache.zeppelin.interpreter.InterpreterSetting.closeAndRemoveAllInterpreterGroups(InterpreterSetting.java:267)
      at org.apache.zeppelin.interpreter.InterpreterFactory.restart(InterpreterFactory.java:1002)
      - locked <0x00000000c0407608> (a java.util.HashMap)
      at org.apache.zeppelin.interpreter.InterpreterFactory.restart(InterpreterFactory.java:1014)
      at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:886)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      - locked <0x00000000c0b727f0> (a java.lang.Object)

      "process reaper" #33 daemon prio=10 os_prio=0 tid=0x00007f1b28f85800 nid=0x143eb runnable [0x00007f1b0009a000]
         java.lang.Thread.State: RUNNABLE
      at java.lang.UNIXProcess.waitForProcessExit(Native Method)
      at java.lang.UNIXProcess.lambda$initStreams$273(UNIXProcess.java:290)
      at java.lang.UNIXProcess$$Lambda$7/1733581655.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "org.eclipse.jetty.server.session.HashSessionManager@5e82df6aTimer" #32 daemon prio=5 os_prio=0 tid=0x00007f1b28da6000 nid=0x143e9 waiting on condition [0x00007f1adf7f6000]
         java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for <0x00000000c0387dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
      at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-31-selector-ServerConnectorManager@65af095e/0" #31 prio=5 os_prio=0 tid=0x00007f1b28d80800 nid=0x143e7 runnable [0x00007f1adf8f7000]
         java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked <0x00000000c12e5048> (a sun.nio.ch.Util$2)
      - locked <0x00000000c12e5038> (a java.util.Collections$UnmodifiableSet)
      - locked <0x00000000c12e4de0> (a sun.nio.ch.EPollSelectorImpl)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
      at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-30-acceptor-2@41ceb414-ServerConnector@608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #30 prio=5 os_prio=0 tid=0x00007f1b28d7f000 nid=0x143e6 waiting for monitor entry [0x00007f1adf9f8000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
      - waiting to lock <0x00000000c1274f50> (a java.lang.Object)
      at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
      at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-29-acceptor-1@307d3332-ServerConnector@608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #29 prio=5 os_prio=0 tid=0x00007f1b28d7d000 nid=0x143e5 waiting for monitor entry [0x00007f1adfaf9000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
      - waiting to lock <0x00000000c1274f50> (a java.lang.Object)
      at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
      at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-28-acceptor-0@3ce9ed8a-ServerConnector@608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #28 prio=5 os_prio=0 tid=0x00007f1b28d7b800 nid=0x143e4 runnable [0x00007f1adfbfa000]
         java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
      - locked <0x00000000c1274f50> (a java.lang.Object)
      at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
      at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-27-acceptor-3@4171a76-ServerConnector@608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #27 prio=5 os_prio=0 tid=0x00007f1b28d79800 nid=0x143e3 waiting for monitor entry [0x00007f1adfcfb000]
         java.lang.Thread.State: BLOCKED (on object monitor)
      at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
      - waiting to lock <0x00000000c1274f50> (a java.lang.Object)
      at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
      at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-26-selector-ServerConnectorManager@65af095e/2" #26 prio=5 os_prio=0 tid=0x00007f1b28d77800 nid=0x143e2 runnable [0x00007f1adfdfc000]
         java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked <0x00000000c1279ed8> (a sun.nio.ch.Util$2)
      - locked <0x00000000c1279ec8> (a java.util.Collections$UnmodifiableSet)
      - locked <0x00000000c1278228> (a sun.nio.ch.EPollSelectorImpl)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
      at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-25-selector-ServerConnectorManager@65af095e/1" #25 prio=5 os_prio=0 tid=0x00007f1b28d76800 nid=0x143e1 runnable [0x00007f1adfefd000]
         java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked <0x00000000c127bf48> (a sun.nio.ch.Util$2)
      - locked <0x00000000c127bf38> (a java.util.Collections$UnmodifiableSet)
      - locked <0x00000000c127a298> (a sun.nio.ch.EPollSelectorImpl)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
      at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "qtp759156157-24-selector-ServerConnectorManager@65af095e/3" #24 prio=5 os_prio=0 tid=0x00007f1b28d75000 nid=0x143e0 runnable [0x00007f1adfffe000]
         java.lang.Thread.State: RUNNABLE
      at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
      at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
      at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
      - locked <0x00000000c1277db0> (a sun.nio.ch.Util$2)
      - locked <0x00000000c1277da0> (a java.util.Collections$UnmodifiableSet)
      - locked <0x00000000c1275540> (a sun.nio.ch.EPollSelectorImpl)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
      at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600)
      at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549)
      at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      at java.lang.Thread.run(Thread.java:745)

      "GC Daemon" #21 daemon prio=2 os_prio=0 tid=0x00007f1b28b13000 nid=0x143de in Object.wait() [0x00007f1b00a11000]
         java.lang.Thread.State: TIMED_WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      - waiting on <0x00000000c001e748> (a sun.misc.GC$LatencyLock)
      at sun.misc.GC$Daemon.run(GC.java:117)
      - locked <0x00000000c001e748> (a sun.misc.GC$LatencyLock)

      "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f1b2811b000 nid=0x143dc runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f1b2810d800 nid=0x143db waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f1b2810b800 nid=0x143da waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f1b28109000 nid=0x143d9 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f1b28107000 nid=0x143d8 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f1b28105000 nid=0x143d7 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f1b28102800 nid=0x143d6 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f1b28100800 nid=0x143d5 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f1b280fe000 nid=0x143d4 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f1b280fc000 nid=0x143d3 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f1b280f9800 nid=0x143d2 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f1b280ef800 nid=0x143d1 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f1b280ed000 nid=0x143d0 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f1b280eb000 nid=0x143cf waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f1b280e9000 nid=0x143ce waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f1b280e6000 nid=0x143cd waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f1b280e4800 nid=0x143cc runnable [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE

      "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1b280ac800 nid=0x143cb in Object.wait() [0x00007f1b024c2000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
      - locked <0x00000000c00268f0> (a java.lang.ref.ReferenceQueue$Lock)
      at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

      "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1b280aa800 nid=0x143ca in Object.wait() [0x00007f1b025c3000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:502)
      at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
      - locked <0x00000000c0042568> (a java.lang.ref.Reference$Lock)

      "main" #1 prio=5 os_prio=0 tid=0x00007f1b2800f000 nid=0x143b1 in Object.wait() [0x00007f1b3161b000]
         java.lang.Thread.State: WAITING (on object monitor)
      at java.lang.Object.wait(Native Method)
      at java.lang.Object.wait(Object.java:502)
      at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:381)
      - locked <0x00000000c009c140> (a java.lang.Object)
      at org.eclipse.jetty.server.Server.join(Server.java:556)
      at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:212)

      "VM Thread" os_prio=0 tid=0x00007f1b280a5800 nid=0x143c9 runnable

      "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f1b28024800 nid=0x143b2 runnable

      "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f1b28026000 nid=0x143b3 runnable

      "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f1b28028000 nid=0x143b4 runnable

      "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f1b2802a000 nid=0x143b5 runnable

      "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f1b2802b800 nid=0x143b6 runnable

      "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f1b2802d800 nid=0x143b7 runnable

      "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f1b2802f800 nid=0x143b8 runnable

      "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f1b28031000 nid=0x143b9 runnable

      "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f1b28033000 nid=0x143ba runnable

      "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f1b28035000 nid=0x143bb runnable

      "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f1b28036800 nid=0x143bc runnable

      "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f1b28038800 nid=0x143bd runnable

      "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f1b2803a800 nid=0x143be runnable

      "GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f1b2803c000 nid=0x143bf runnable

      "GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f1b2803e000 nid=0x143c0 runnable

      "GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f1b2803f800 nid=0x143c1 runnable

      "GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f1b28041800 nid=0x143c2 runnable

      "GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f1b28043800 nid=0x143c3 runnable

      "GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f1b28045000 nid=0x143c4 runnable

      "GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f1b28047000 nid=0x143c5 runnable

      "GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f1b28049000 nid=0x143c6 runnable

      "GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f1b2804a800 nid=0x143c7 runnable

      "GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f1b2804c800 nid=0x143c8 runnable

      "VM Periodic Task Thread" os_prio=0 tid=0x00007f1b2811e000 nid=0x143dd waiting on condition

      JNI global references: 289

      Show
      2017-03-27 16:27:06 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.60-b23 mixed mode): "Attach Listener" #96 daemon prio=9 os_prio=0 tid=0x00007f1a80001000 nid=0x14576 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "qtp759156157-95" #95 prio=5 os_prio=0 tid=0x00007f19c0002000 nid=0x14543 waiting on condition [0x00007f1add2fe000]    java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00a2de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp759156157-94" #94 prio=5 os_prio=0 tid=0x00007f19c0001000 nid=0x14542 waiting for monitor entry [0x00007f1add0f9000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317) at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:241) at org.apache.zeppelin.notebook.Paragraph.getCurrentRepl(Paragraph.java:245) at org.apache.zeppelin.helium.Helium.suggestApp(Helium.java:249) at org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:84) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180) at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:205) at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:102) at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94) at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) - locked <0x00000000ee0be198> (a org.apache.cxf.phase.PhaseInterceptorChain) at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239) at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222) at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153) at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:211) at javax.servlet.http.HttpServlet.service(HttpServlet.java:575) at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.apache.zeppelin.server.CorsFilter.doFilter(CorsFilter.java:72) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:499) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "pool-2-thread-4" #93 prio=5 os_prio=0 tid=0x00007f1990001000 nid=0x14513 waiting on condition [0x00007f1adc858000]    java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "pool-2-thread-3" #91 prio=5 os_prio=0 tid=0x00007f1998001000 nid=0x14511 waiting on condition [0x00007f1ad7ffe000]    java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "pool-4-thread-1" #85 prio=5 os_prio=0 tid=0x00007f19a0001800 nid=0x1449f waiting on condition [0x00007f1adca5a000]    java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000eb62fe18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zeppelin.interpreter.remote.AppendOutputRunner.run(AppendOutputRunner.java:66) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Thread-41" #77 prio=5 os_prio=0 tid=0x00007f19b802e800 nid=0x1449c waiting for monitor entry [0x00007f1adcb5b000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.get(InterpreterFactory.java:891) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.socket.NotebookServer.onMetaInfosReceived(NotebookServer.java:2181) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventPoller.run(RemoteInterpreterEventPoller.java:241) "Exec Stream Pumper" #84 daemon prio=5 os_prio=0 tid=0x00007f19b4003000 nid=0x14433 runnable [0x00007f1ad7bfa000]    java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000000c126c760> (a java.lang.UNIXProcess$ProcessPipeInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107) at java.lang.Thread.run(Thread.java:745) "Exec Stream Pumper" #83 daemon prio=5 os_prio=0 tid=0x00007f19b4002000 nid=0x14432 runnable [0x00007f1ad7cfb000]    java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000000c121a900> (a java.lang.UNIXProcess$ProcessPipeInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:107) at org.apache.commons.exec.StreamPumper.run(StreamPumper.java:107) at java.lang.Thread.run(Thread.java:745) "Exec Default Executor" #82 prio=5 os_prio=0 tid=0x00007f19b8011000 nid=0x14430 in Object.wait() [0x00007f1ad7dfc000]    java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.UNIXProcess.waitFor(UNIXProcess.java:396) - locked <0x00000000c1265e90> (a java.lang.UNIXProcess) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:364) at org.apache.commons.exec.DefaultExecutor.access$200(DefaultExecutor.java:48) at org.apache.commons.exec.DefaultExecutor$1.run(DefaultExecutor.java:200) at java.lang.Thread.run(Thread.java:745) "Thread-43" #81 prio=5 os_prio=0 tid=0x00007f19b8003800 nid=0x1442f waiting for monitor entry [0x00007f1ad7efd000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317) at org.apache.zeppelin.notebook.Paragraph.getRepl(Paragraph.java:241) at org.apache.zeppelin.notebook.Paragraph.progress(Paragraph.java:309) at org.apache.zeppelin.scheduler.JobProgressPoller.run(JobProgressPoller.java:51) "pool-2-thread-2" #79 prio=5 os_prio=0 tid=0x00007f19c4001800 nid=0x1442d waiting for monitor entry [0x00007f1adc483000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.notebook.Note.snapshotAngularObjectRegistry(Note.java:624) at org.apache.zeppelin.notebook.Note.persist(Note.java:677) at org.apache.zeppelin.socket.NotebookServer$ParagraphListenerImpl.afterStatusChange(NotebookServer.java:2032) at org.apache.zeppelin.scheduler.Job.setStatus(Job.java:149) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:349) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "pool-2-thread-1" #78 prio=5 os_prio=0 tid=0x00007f19c8021800 nid=0x1442c waiting on condition [0x00007f1adc585000]    java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0366148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "qtp759156157-76" #76 prio=5 os_prio=0 tid=0x00007f19cc001000 nid=0x1442b waiting on condition [0x00007f1adcefa000]    java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c00a2de8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:531) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:47) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:590) at java.lang.Thread.run(Thread.java:745) "qtp759156157-75" #75 prio=5 os_prio=0 tid=0x00007f19c800a800 nid=0x1442a waiting for monitor entry [0x00007f1adcffa000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreter(InterpreterFactory.java:1317) at org.apache.zeppelin.notebook.Paragraph.isValidInterpreter(Paragraph.java:673) at org.apache.zeppelin.notebook.Paragraph.getMagic(Paragraph.java:662) at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1607) at org.apache.zeppelin.socket.NotebookServer.runParagraph(NotebookServer.java:1600) at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:263) at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128) at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65) at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161) at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309) at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214) at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220) at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "Thread-37" #73 daemon prio=5 os_prio=0 tid=0x00007f19dc013800 nid=0x14428 waiting on condition [0x00007f1add1fd000]    java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c124c050> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Scheduler-1566502717" #71 prio=5 os_prio=0 tid=0x00007f1a1c002000 nid=0x14425 waiting on condition [0x00007f1add500000]    java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c1266320> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "Timer-0" #67 daemon prio=5 os_prio=0 tid=0x00007f1b2ae81800 nid=0x14420 in Object.wait() [0x00007f1ade4c7000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.util.TimerThread.mainLoop(Timer.java:552) - locked <0x00000000c0ad9a68> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:505) "DefaultQuartzScheduler_QuartzSchedulerThread" #66 prio=5 os_prio=0 tid=0x00007f1b2ae7e800 nid=0x1441f in Object.wait() [0x00007f1ade5c8000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:410) - locked <0x00000000c0b64ac0> (a java.lang.Object) "DefaultQuartzScheduler_Worker-10" #65 prio=5 os_prio=0 tid=0x00007f1b28125000 nid=0x1441e in Object.wait() [0x00007f1ade6c9000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b72278> (a java.lang.Object) "DefaultQuartzScheduler_Worker-9" #64 prio=5 os_prio=0 tid=0x00007f1b28123000 nid=0x1441d in Object.wait() [0x00007f1ade7ca000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b7db50> (a java.lang.Object) "DefaultQuartzScheduler_Worker-8" #63 prio=5 os_prio=0 tid=0x00007f1b28121000 nid=0x1441c in Object.wait() [0x00007f1ade9cc000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b74000> (a java.lang.Object) "DefaultQuartzScheduler_Worker-7" #62 prio=5 os_prio=0 tid=0x00007f1b2813c000 nid=0x1441b in Object.wait() [0x00007f1ad7361000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b73c90> (a java.lang.Object) "DefaultQuartzScheduler_Worker-6" #61 prio=5 os_prio=0 tid=0x00007f1b2813a800 nid=0x1441a in Object.wait() [0x00007f1adc2b1000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b73920> (a java.lang.Object) "DefaultQuartzScheduler_Worker-5" #60 prio=5 os_prio=0 tid=0x00007f1b28139800 nid=0x14419 in Object.wait() [0x00007f1ade8cb000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568) - locked <0x00000000c0b735b0> (a java.lang.Object) "DefaultQuartzScheduler_Worker-4" #59 prio=5 os_prio=0 tid=0x00007f1b28138000 nid=0x14418 waiting for monitor entry [0x00007f1adc1b0000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0b73240> (a java.lang.Object) "DefaultQuartzScheduler_Worker-3" #58 prio=5 os_prio=0 tid=0x00007f1b28137000 nid=0x14417 waiting for monitor entry [0x00007f1ad715f000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0b72ed0> (a java.lang.Object) "DefaultQuartzScheduler_Worker-2" #57 prio=5 os_prio=0 tid=0x00007f1b289e0800 nid=0x14416 waiting for monitor entry [0x00007f1ad705e000]    java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.zeppelin.interpreter.InterpreterFactory.getNoteInterpreterSettingBinding(InterpreterFactory.java:932) - waiting to lock <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.getInterpreterSettings(InterpreterFactory.java:1199) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:885) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0b72b60> (a java.lang.Object) "DefaultQuartzScheduler_Worker-1" #56 prio=5 os_prio=0 tid=0x00007f1b289e1800 nid=0x14415 runnable [0x00007f1ad725f000]    java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked <0x00000000eb696aa8> (a java.io.BufferedInputStream) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.recv_shutdown(RemoteInterpreterService.java:385) at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Client.shutdown(RemoteInterpreterService.java:373) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.dereference(RemoteInterpreterProcess.java:128) - locked <0x00000000c12567a8> (a java.util.concurrent.atomic.AtomicInteger) at org.apache.zeppelin.interpreter.InterpreterGroup.close(InterpreterGroup.java:155) at org.apache.zeppelin.interpreter.InterpreterSetting.closeAndRemoveInterpreterGroupByNoteId(InterpreterSetting.java:238) at org.apache.zeppelin.interpreter.InterpreterSetting.closeAndRemoveAllInterpreterGroups(InterpreterSetting.java:267) at org.apache.zeppelin.interpreter.InterpreterFactory.restart(InterpreterFactory.java:1002) - locked <0x00000000c0407608> (a java.util.HashMap) at org.apache.zeppelin.interpreter.InterpreterFactory.restart(InterpreterFactory.java:1014) at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:886) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) - locked <0x00000000c0b727f0> (a java.lang.Object) "process reaper" #33 daemon prio=10 os_prio=0 tid=0x00007f1b28f85800 nid=0x143eb runnable [0x00007f1b0009a000]    java.lang.Thread.State: RUNNABLE at java.lang.UNIXProcess.waitForProcessExit(Native Method) at java.lang.UNIXProcess.lambda$initStreams$273(UNIXProcess.java:290) at java.lang.UNIXProcess$$Lambda$7/1733581655.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) " org.eclipse.jetty.server.session.HashSessionManager@5e82df6aTimer " #32 daemon prio=5 os_prio=0 tid=0x00007f1b28da6000 nid=0x143e9 waiting on condition [0x00007f1adf7f6000]    java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000c0387dd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) " qtp759156157-31-selector-ServerConnectorManager@65af095e /0" #31 prio=5 os_prio=0 tid=0x00007f1b28d80800 nid=0x143e7 runnable [0x00007f1adf8f7000]    java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c12e5048> (a sun.nio.ch.Util$2) - locked <0x00000000c12e5038> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c12e4de0> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-30-acceptor-2@41ceb414-ServerConnector @608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #30 prio=5 os_prio=0 tid=0x00007f1b28d7f000 nid=0x143e6 waiting for monitor entry [0x00007f1adf9f8000]    java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234) - waiting to lock <0x00000000c1274f50> (a java.lang.Object) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-29-acceptor-1@307d3332-ServerConnector @608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #29 prio=5 os_prio=0 tid=0x00007f1b28d7d000 nid=0x143e5 waiting for monitor entry [0x00007f1adfaf9000]    java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234) - waiting to lock <0x00000000c1274f50> (a java.lang.Object) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-28-acceptor-0@3ce9ed8a-ServerConnector @608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #28 prio=5 os_prio=0 tid=0x00007f1b28d7b800 nid=0x143e4 runnable [0x00007f1adfbfa000]    java.lang.Thread.State: RUNNABLE at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked <0x00000000c1274f50> (a java.lang.Object) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-27-acceptor-3@4171a76-ServerConnector @608b1fd2{HTTP/1.1}{0.0.0.0:9095}" #27 prio=5 os_prio=0 tid=0x00007f1b28d79800 nid=0x143e3 waiting for monitor entry [0x00007f1adfcfb000]    java.lang.Thread.State: BLOCKED (on object monitor) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234) - waiting to lock <0x00000000c1274f50> (a java.lang.Object) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-26-selector-ServerConnectorManager@65af095e /2" #26 prio=5 os_prio=0 tid=0x00007f1b28d77800 nid=0x143e2 runnable [0x00007f1adfdfc000]    java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c1279ed8> (a sun.nio.ch.Util$2) - locked <0x00000000c1279ec8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c1278228> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-25-selector-ServerConnectorManager@65af095e /1" #25 prio=5 os_prio=0 tid=0x00007f1b28d76800 nid=0x143e1 runnable [0x00007f1adfefd000]    java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c127bf48> (a sun.nio.ch.Util$2) - locked <0x00000000c127bf38> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c127a298> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) " qtp759156157-24-selector-ServerConnectorManager@65af095e /3" #24 prio=5 os_prio=0 tid=0x00007f1b28d75000 nid=0x143e0 runnable [0x00007f1adfffe000]    java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000c1277db0> (a sun.nio.ch.Util$2) - locked <0x00000000c1277da0> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000c1275540> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:600) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:549) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) at java.lang.Thread.run(Thread.java:745) "GC Daemon" #21 daemon prio=2 os_prio=0 tid=0x00007f1b28b13000 nid=0x143de in Object.wait() [0x00007f1b00a11000]    java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <0x00000000c001e748> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:117) - locked <0x00000000c001e748> (a sun.misc.GC$LatencyLock) "Service Thread" #20 daemon prio=9 os_prio=0 tid=0x00007f1b2811b000 nid=0x143dc runnable [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C1 CompilerThread14" #19 daemon prio=9 os_prio=0 tid=0x00007f1b2810d800 nid=0x143db waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C1 CompilerThread13" #18 daemon prio=9 os_prio=0 tid=0x00007f1b2810b800 nid=0x143da waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C1 CompilerThread12" #17 daemon prio=9 os_prio=0 tid=0x00007f1b28109000 nid=0x143d9 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C1 CompilerThread11" #16 daemon prio=9 os_prio=0 tid=0x00007f1b28107000 nid=0x143d8 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C1 CompilerThread10" #15 daemon prio=9 os_prio=0 tid=0x00007f1b28105000 nid=0x143d7 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread9" #14 daemon prio=9 os_prio=0 tid=0x00007f1b28102800 nid=0x143d6 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread8" #13 daemon prio=9 os_prio=0 tid=0x00007f1b28100800 nid=0x143d5 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread7" #12 daemon prio=9 os_prio=0 tid=0x00007f1b280fe000 nid=0x143d4 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread6" #11 daemon prio=9 os_prio=0 tid=0x00007f1b280fc000 nid=0x143d3 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread5" #10 daemon prio=9 os_prio=0 tid=0x00007f1b280f9800 nid=0x143d2 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread4" #9 daemon prio=9 os_prio=0 tid=0x00007f1b280ef800 nid=0x143d1 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread3" #8 daemon prio=9 os_prio=0 tid=0x00007f1b280ed000 nid=0x143d0 waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f1b280eb000 nid=0x143cf waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f1b280e9000 nid=0x143ce waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f1b280e6000 nid=0x143cd waiting on condition [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f1b280e4800 nid=0x143cc runnable [0x0000000000000000]    java.lang.Thread.State: RUNNABLE "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f1b280ac800 nid=0x143cb in Object.wait() [0x00007f1b024c2000]    java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) - locked <0x00000000c00268f0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f1b280aa800 nid=0x143ca in Object.wait() [0x00007f1b025c3000]    java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157) - locked <0x00000000c0042568> (a java.lang.ref.Reference$Lock) "main" #1 prio=5 os_prio=0 tid=0x00007f1b2800f000 nid=0x143b1 in Object.wait() [0x00007f1b3161b000]    java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:381) - locked <0x00000000c009c140> (a java.lang.Object) at org.eclipse.jetty.server.Server.join(Server.java:556) at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:212) "VM Thread" os_prio=0 tid=0x00007f1b280a5800 nid=0x143c9 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f1b28024800 nid=0x143b2 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f1b28026000 nid=0x143b3 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f1b28028000 nid=0x143b4 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f1b2802a000 nid=0x143b5 runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007f1b2802b800 nid=0x143b6 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007f1b2802d800 nid=0x143b7 runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x00007f1b2802f800 nid=0x143b8 runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x00007f1b28031000 nid=0x143b9 runnable "GC task thread#8 (ParallelGC)" os_prio=0 tid=0x00007f1b28033000 nid=0x143ba runnable "GC task thread#9 (ParallelGC)" os_prio=0 tid=0x00007f1b28035000 nid=0x143bb runnable "GC task thread#10 (ParallelGC)" os_prio=0 tid=0x00007f1b28036800 nid=0x143bc runnable "GC task thread#11 (ParallelGC)" os_prio=0 tid=0x00007f1b28038800 nid=0x143bd runnable "GC task thread#12 (ParallelGC)" os_prio=0 tid=0x00007f1b2803a800 nid=0x143be runnable "GC task thread#13 (ParallelGC)" os_prio=0 tid=0x00007f1b2803c000 nid=0x143bf runnable "GC task thread#14 (ParallelGC)" os_prio=0 tid=0x00007f1b2803e000 nid=0x143c0 runnable "GC task thread#15 (ParallelGC)" os_prio=0 tid=0x00007f1b2803f800 nid=0x143c1 runnable "GC task thread#16 (ParallelGC)" os_prio=0 tid=0x00007f1b28041800 nid=0x143c2 runnable "GC task thread#17 (ParallelGC)" os_prio=0 tid=0x00007f1b28043800 nid=0x143c3 runnable "GC task thread#18 (ParallelGC)" os_prio=0 tid=0x00007f1b28045000 nid=0x143c4 runnable "GC task thread#19 (ParallelGC)" os_prio=0 tid=0x00007f1b28047000 nid=0x143c5 runnable "GC task thread#20 (ParallelGC)" os_prio=0 tid=0x00007f1b28049000 nid=0x143c6 runnable "GC task thread#21 (ParallelGC)" os_prio=0 tid=0x00007f1b2804a800 nid=0x143c7 runnable "GC task thread#22 (ParallelGC)" os_prio=0 tid=0x00007f1b2804c800 nid=0x143c8 runnable "VM Periodic Task Thread" os_prio=0 tid=0x00007f1b2811e000 nid=0x143dd waiting on condition JNI global references: 289

      Description

      Cannot execute other notebook when some notebook config cronjob cause request of '/api/helium/suggest/<noteid>/<Paragraph ID>' response with timeout. java stack trace is bellow:

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rockiee281 richard.lee
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: