[TEZ-3967] DAGImpl: dag lock is unfair and can starve the writers

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: László Bodor
• Reporter: Gopal Vijayaraghavan
• Created: 06/Jul/18 08:11
• Updated: 08/May/20 08:04

Description

Found when debugging HIVE-20103, that a reader arriving when another reader is active can postpone a writer from obtaining a write-lock.

This is fundamentally bad for the DAGImpl as useful progress can only happen when the writeLock is held.

  public void handle(DAGEvent event) {
...
    try {
      writeLock.lock();

   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007efb02246f40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:1162)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:149)
        at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2251)
        at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2242)
        at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
        at java.lang.Thread.run(Thread.java:745)

while read-lock is passed around between

       at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
        at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
        at org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)

calls.

Issue Links

• is related to: HIVE-20103 WM: Only Aggregate DAG counters if at least one is used