[DRILL-1929] After canceling query in sqlline subsequent query in same session hangs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0
• Fix Version/s: 1.2.0
• Component/s: Execution - RPC
• Assignee: Jacques Nadeau
• Reporter: Aman Sinha
• Created: 05/Jan/15 20:10
• Updated: 30/Sep/15 00:39
• Resolved: 11/Sep/15 23:25

Description

This might be related to existing query cancel issues but I did not find an exact match. Run the following query (against Tpc-ds schema) in sqlline and cntr-c to cancel it after couple of seconds:

select ss_sold_date_sk, ss_item_sk, ss_cdemo_sk, ss_hdemo_sk, ss_store_sk, ss_list_price from store_sales;

The query gets cancelled successfully. However, when I submit the same query again in the same session the query hangs. Here's a relevant portion of the jstack trace:

"2b55158d-6837-71ad-d523-bee5055ccc78:frag:0:0" daemon prio=5 tid=0x00007f8d2c4d0000 nid=0x572b waiting on condition [0x0000000122997000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000703481ba8> (a org.apache.drill.exec.rpc.ResettableBarrier$InternalSynchronizer)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
        at org.apache.drill.exec.rpc.ResettableBarrier.await(ResettableBarrier.java:70)
        at org.apache.drill.exec.rpc.RemoteConnection$WriteManager.waitForWritable(RemoteConnection.java:80)
        at org.apache.drill.exec.rpc.RemoteConnection.blockOnNotWritable(RemoteConnection.java:53)
        at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:91)
        at org.apache.drill.exec.rpc.user.UserServer$UserClientConnection.sendResult(UserServer.java:123)
        at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:157)
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57)
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114)
        at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254)

So, it looks like the WriteManager is blocked on a non-writable Rpc channel, which indicates the query cancellation before did not release all resources.

Issue Links

• depends upon: DRILL-2245 Improve Drillbit resiliency
• is part of: DRILL-2755 Use and handle InterruptedException during query processing