[TWILL-110] Deadlock when shutting down runnable container

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.4.0-incubating
• Fix Version/s: 0.5.0-incubating
• Component/s: core
• Assignee: Terence Yim
• Reporter: Terence Yim
• Created: 26/Nov/14 20:48
• Updated: 02/Dec/14 22:11
• Resolved: 02/Dec/14 22:11

Description

Deadlock was observed when a TwillRunnable container was shutting down, causing the process hanging.

Found one Java-level deadlock:
=============================
"Thread-5":
  waiting for ownable synchronizer 0x00000000eb1a6ec0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "zk-client-EventThread"
"zk-client-EventThread":
  waiting for ownable synchronizer 0x00000000eb18c698, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "Thread-5"

Java stack information for the threads listed above:
===================================================
"Thread-5":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000eb1a6ec0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:700)
at java.util.concurrent.ThreadPoolExecutor.shutdown(ThreadPoolExecutor.java:1397)
at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.doStop(DefaultZKClientService.java:402)
at com.google.common.util.concurrent.AbstractService.stop(AbstractService.java:198)
at org.apache.twill.internal.zookeeper.DefaultZKClientService.stop(DefaultZKClientService.java:310)
at org.apache.twill.zookeeper.ForwardingZKClientService.stop(ForwardingZKClientService.java:66)
at org.apache.twill.common.Services$2.run(Services.java:131)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:262)
at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
at com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:105)
at com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:160)
at org.apache.twill.common.Services.doChain(Services.java:106)
at org.apache.twill.common.Services.chainStop(Services.java:61)
at org.apache.twill.internal.ServiceMain$1.run(ServiceMain.java:69)
"zk-client-EventThread":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000eb18c698> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
at com.google.common.util.concurrent.AbstractService.notifyStopped(AbstractService.java:267)
at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate.access$600(DefaultZKClientService.java:372)
at org.apache.twill.internal.zookeeper.DefaultZKClientService$ServiceDelegate$1.terminated(DefaultZKClientService.java:382)
at java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:704)
at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1006)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Found 1 deadlock.