Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-2502

RemoteInterpreterServer hang forever during shutdown

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.7.0, 0.7.1, 0.7.2, 0.8.0
    • 0.7.2, 0.8.0
    • zeppelin-interpreter
    • None

    Description

      TL;DR;

      there is the opportunity that RemoteServerInterpreter does never terminate during shutdown causing zombies JVMs around.

      a proper Unit test is provided here:
      https://github.com/nokia/zeppelin/blob/35161ba8595782275a1f597f9267a92c226a6cc9/zeppelin-interpreter/src/test/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServerTest.java#L92

      VERSION:

      bug detected on 0.7.0 but applicable on master

      DETAILED:

      I was digging a problem with zombie JVM process caused by the interaction of long running Spark jobs in the cluster and Spark interpreter restarts done via UI.
      I managed to reproduce a scenario where a SparkSubmit process never ends, and analyzing the stack trace I ended up finding potential conditions where I can have zombies in RemoteInterpreter.

      I'm still not 100% sure this will solve but it is anyhow a fix to a bug that can manifest itself sooner or later.

      My stack trace of the hanging process was:
      Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode):
      "Attach Listener" #118 daemon prio=9 os_prio=0 tid=0x00007fb03c001000 nid=0x104e waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
      Locked ownable synchronizers:

      • None
        "pool-2-thread-6" #114 prio=5 os_prio=0 tid=0x00007fb00c004000 nid=0xe52 waiting on condition [0x00007faf963c1000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000c00da470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "pool-2-thread-5" #108 prio=5 os_prio=0 tid=0x00007fb01403d000 nid=0xd8d waiting on condition [0x00007faf966c4000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000c00da470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "pool-2-thread-4" #107 prio=5 os_prio=0 tid=0x00007fb01403c000 nid=0xd8c in Object.wait() [0x00007faf967c5000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
      • locked <0x00000000dc206700> (a java.util.LinkedList)
      • locked <0x00000000dc206700> (a java.util.LinkedList)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000db13c610> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "pool-2-thread-3" #105 prio=5 os_prio=0 tid=0x00007fb00c003000 nid=0xd8a in Object.wait() [0x00007fafa7cfd000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
      • locked <0x00000000dc206838> (a java.util.LinkedList)
      • locked <0x00000000dc206838> (a java.util.LinkedList)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000db13c7f8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #90 daemon prio=5 os_prio=0 tid=0x00007fb00af63000 nid=0xccd in Object.wait() [0x00007faf9cfe1000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
      • locked <0x00000000cce05198> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3063)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "BoneCP-pool-watch-thread" #89 daemon prio=5 os_prio=0 tid=0x00007fb009fdd000 nid=0xccc waiting on condition [0x00007faf9d8e2000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000cd439380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000cd439468> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "BoneCP-keep-alive-scheduler" #88 daemon prio=5 os_prio=0 tid=0x00007fb009fdb800 nid=0xccb waiting on condition [0x00007faf9d9e3000]
        java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000cd43fdf0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "com.google.common.base.internal.Finalizer" #87 daemon prio=5 os_prio=0 tid=0x00007fb009fda000 nid=0xcca in Object.wait() [0x00007faf9dae4000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
      • locked <0x00000000cd4881a0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at com.google.common.base.internal.Finalizer.run(Finalizer.java:132)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "BoneCP-pool-watch-thread" #86 daemon prio=5 os_prio=0 tid=0x00007fb00ad1f000 nid=0xcc9 waiting on condition [0x00007faf9dfe5000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000cd4891e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000cd4892c8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "BoneCP-keep-alive-scheduler" #85 daemon prio=5 os_prio=0 tid=0x00007fb00ad21000 nid=0xcc8 waiting on condition [0x00007faf9e0e6000]
        java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000cd489c18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "com.google.common.base.internal.Finalizer" #84 daemon prio=5 os_prio=0 tid=0x00007fb00ad1d000 nid=0xcc7 in Object.wait() [0x00007faf9e1e7000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
      • locked <0x00000000cd48a528> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at com.google.common.base.internal.Finalizer.run(Finalizer.java:132)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "derby.rawStoreDaemon" #83 daemon prio=5 os_prio=0 tid=0x00007fb00acb8800 nid=0xcc6 in Object.wait() [0x00007faf9e4e8000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
      • locked <0x00000000cceb07f8> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "Timer-1" #82 daemon prio=5 os_prio=0 tid=0x00007fb00ac4f800 nid=0xcc5 in Object.wait() [0x00007faf9e7e9000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
      • locked <0x00000000cd3b9f08> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)
        Locked ownable synchronizers:
      • None
        "pool-1-thread-5" #81 prio=5 os_prio=0 tid=0x00007fb024008800 nid=0xcc1 runnable [0x00007faf9f8ea000]
        java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      • locked <0x00000000cd48ac30> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000cd48ad40> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "pool-2-thread-2" #15 prio=5 os_prio=0 tid=0x00007fb00c002000 nid=0xc72 waiting on condition [0x00007fb013ffe000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000c00da470> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "pool-2-thread-1" #14 prio=5 os_prio=0 tid=0x00007fb0200b4000 nid=0xc71 in Object.wait() [0x00007fb0281ca000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.zeppelin.scheduler.FIFOScheduler.run(FIFOScheduler.java:112)
      • locked <0x00000000c0003a20> (a java.util.LinkedList)
      • locked <0x00000000c0003a20> (a java.util.LinkedList)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000c0003b18> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "pool-1-thread-4" #13 prio=5 os_prio=0 tid=0x00007fb024006800 nid=0xc70 waiting on condition [0x00007fb0286dc000]
        java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000000c0004058> (a java.util.concurrent.SynchronousQueue$TransferStack)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • None
        "pool-1-thread-3" #12 prio=5 os_prio=0 tid=0x00007fb024005800 nid=0xc6f runnable [0x00007fb0287dd000]
        java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      • locked <0x00000000c00ff608> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000c0007180> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "pool-1-thread-2" #11 prio=5 os_prio=0 tid=0x00007fb02400a800 nid=0xc6e in Object.wait() [0x00007fb0288de000]
        java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterEventClient.pollEvent(RemoteInterpreterEventClient.java:220)
      • locked <0x00000000c0003cc0> (a java.util.LinkedList)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.getEvent(RemoteInterpreterServer.java:791)
        at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1863)
        at org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService$Processor$getEvent.getResult(RemoteInterpreterService.java:1848)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000c000d070> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "pool-1-thread-1" #10 prio=5 os_prio=0 tid=0x00007fb024003800 nid=0xc6d runnable [0x00007fb0289df000]
        java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
      • locked <0x00000000d95c4508> (a java.io.BufferedInputStream)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:285)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        Locked ownable synchronizers:
      • <0x00000000c0004210> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        "Thread-0" #9 prio=5 os_prio=0 tid=0x00007fb07cf75800 nid=0xc6a runnable [0x00007fb028ae0000]
        java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:129)
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:60)
        at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:161)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.run(RemoteInterpreterServer.java:96)
        Locked ownable synchronizers:
      • None
        "Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007fb07c0d1000 nid=0xc68 runnable [0x0000000000000000]
        java.lang.Thread.State: RUNNABLE
        Locked ownable synchronizers:
      • None
        "C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007fb07c0bb800 nid=0xc67 waiting on condition [0x0000000000000000]
        java.lang.Thread.State: RUNNABLE
        Locked ownable synchronizers:
      • None
        "C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007fb07c0b9800 nid=0xc66 waiting on condition [0x0000000000000000]
        java.lang.Thread.State: RUNNABLE
        Locked ownable synchronizers:
      • None
        "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007fb07c0b6800 nid=0xc65 waiting on condition [0x0000000000000000]
        java.lang.Thread.State: RUNNABLE
        Locked ownable synchronizers:
      • None
        "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007fb07c0b4800 nid=0xc64 runnable [0x0000000000000000]
        java.lang.Thread.State: RUNNABLE
        Locked ownable synchronizers:
      • None
        "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fb07c086000 nid=0xc63 in Object.wait() [0x00007fb0513f2000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
      • locked <0x00000000c00028f0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
        Locked ownable synchronizers:
      • None
        "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fb07c084000 nid=0xc62 in Object.wait() [0x00007fb0514f3000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
      • locked <0x00000000c00030f0> (a java.lang.ref.Reference$Lock)
        Locked ownable synchronizers:
      • None
        "main" #1 prio=5 os_prio=0 tid=0x00007fb07c00f000 nid=0xc5a in Object.wait() [0x00007fb083f1c000]
        java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
      • waiting on <0x00000000c0003d00> (a org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1245)
      • locked <0x00000000c0003d00> (a org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer)
        at java.lang.Thread.join(Thread.java:1319)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer.main(RemoteInterpreterServer.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
        Locked ownable synchronizers:
      • None
        "VM Thread" os_prio=0 tid=0x00007fb07c07f000 nid=0xc61 runnable
        "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fb07c024800 nid=0xc5b runnable
        "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007fb07c026000 nid=0xc5c runnable
        "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007fb07c028000 nid=0xc5d runnable
        "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007fb07c02a000 nid=0xc5e runnable
        "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x00007fb07c02b800 nid=0xc5f runnable
        "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x00007fb07c02d800 nid=0xc60 runnable
        "VM Periodic Task Thread" os_prio=0 tid=0x00007fb07c0db800 nid=0xc69 waiting on condition
        JNI global references: 288

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #2322

          Activity

            People

              andreatp Andrea Peruffo
              andreatp Andrea Peruffo
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: