Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3967

DAGImpl: dag lock is unfair and can starve the writers

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Found when debugging HIVE-20103, that a reader arriving when another reader is active can postpone a writer from obtaining a write-lock.

      This is fundamentally bad for the DAGImpl as useful progress can only happen when the writeLock is held.

        public void handle(DAGEvent event) {
      ...
          try {
            writeLock.lock();
      
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00007efb02246f40> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
              at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:1162)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.handle(DAGImpl.java:149)
              at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2251)
              at org.apache.tez.dag.app.DAGAppMaster$DagEventDispatcher.handle(DAGAppMaster.java:2242)
              at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)
              at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:115)
              at java.lang.Thread.run(Thread.java:745)
      

      while read-lock is passed around between

             at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:901)
              at org.apache.tez.dag.app.dag.impl.DAGImpl.getDAGStatus(DAGImpl.java:940)
              at org.apache.tez.dag.api.client.DAGClientHandler.getDAGStatus(DAGClientHandler.java:73)
      

      calls.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gopalv Gopal V
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: