[TEZ-493] Task JVMs hanging around even after the AM/app has completed

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Assignee: Hitesh Shah
• Reporter: Siddharth Seth
• Created: 24/Sep/13 22:59
• Updated: 01/Dec/13 20:21
• Resolved: 01/Oct/13 23:03

Description

"TezTaskEventRouter[attempt_1380003134543_0003_1_00_000000_1]" prio=5 tid=7ff149477800 nid=0x10ef34000 waiting on condition [10ef33000]
   java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <7f00c9148> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.apache.tez.engine.newruntime.LogicalIOProcessorRuntimeTask$1.run(LogicalIOProcessorRuntimeTask.java:407)
  at java.lang.Thread.run(Thread.java:680)

That's one trace, from a process started on 09/23. Don't think any fixes went in after that, which would resolve this.

Saw additional stuck processes because the Inputs/Ouputs created non-daemon threads (which also happened to be blocking on a queue)