[TEZ-2362] State Change Notifier Thread should be stopped when dag is completed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Assignee: Bikas Saha
• Reporter: Jeff Zhang
• Created: 24/Apr/15 03:16
• Updated: 30/Jun/15 04:52
• Resolved: 27/Apr/15 17:12

Description

For each dag, there's one thread of State Change Notifier. Should stop it when dag is completed. Otherwise it may exhaust the thread resources of AM and slow down AM for one sessions with lots of DAGs.

JStack trace for multiple threads of State Change Notifier

"State Change Notifier DAG: dag_1429844178238_0001_3" daemon prio=5 tid=0x00007fa525596800 nid=0x102503 waiting on condition [0x000000019e362000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000706f2b3e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tez.dag.app.dag.StateChangeNotifier$1.run(StateChangeNotifier.java:100)

   Locked ownable synchronizers:
    - None

"State Change Notifier DAG: dag_1429844178238_0001_2" daemon prio=5 tid=0x00007fa529025000 nid=0x101f03 waiting on condition [0x000000019e059000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000706f2f3e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tez.dag.app.dag.StateChangeNotifier$1.run(StateChangeNotifier.java:100)

   Locked ownable synchronizers:
    - None

"State Change Notifier DAG: dag_1429844178238_0001_1" daemon prio=5 tid=0x00007fa52539a800 nid=0xd1133 waiting on condition [0x00000001859d3000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000706f2d500> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tez.dag.app.dag.StateChangeNotifier$1.run(StateChangeNotifier.java:100)

   Locked ownable synchronizers:
    - None

"State Change Notifier DAG: dag_1429844178013_0001_1" daemon prio=5 tid=0x00007fa526396800 nid=0xd191b waiting on condition [0x0000000192a3d000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x0000000706f18c98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    at org.apache.tez.dag.app.dag.StateChangeNotifier$1.run(StateChangeNotifier.java:100)

   Locked ownable synchronizers:
    - None