Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4330

Long running SQL query hangs once Foreman node is killed

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.0
    • None
    • Execution - Flow
    • None
    • 4 node cluster CentOS

    Description

      Summary : Once Foreman node Drillbit is killed, long running query just hangs and no profile information is written to Web UI. That long running query was issued from the Foreman node.

      MapR Drill 1.4.0 GA
      MapR FS 5.0.0 GA
      JDK8
      4 node CentOS cluster

      ./sqlline -u "jdbc:drill:schema=dfs.tmp -n mapr -p mapr"
      Issue a long running select query over JSON data
      Immediately kill the Drillbit on Foreman node (ps -eaf | grep Drillbit), kill -9 PID

      The long running query hangs on sqlline prompt, there are no messages/errors/Exceptions reported on sqlline prompt.

      On the Web UI there is no profile reported for the long running query that was running on the Drillbit that was killed.

      Question (1) : Why was there no profile reported/written on the Web UI for that long running query ? In a real production scenario user will not know what query was under execution at the point when Foreman went down.

      Question (2) : Why does the long running query not terminate, once the foreman was killed ? from the drillbit.log snippet we do not see any CANCELED/TERMINATED message for that query, why ?

      Snippet from drillbit.log on the foreman node.

      2016-02-01 10:59:20,917 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.drill.exec.work.foreman.Foreman - Query text for query id 2950c576-b2d2-5bc3-e9b5-ff4414d088c0: select * from `twoKeyJsn.json`
      2016-02-01 10:59:21,067 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.create() took 1 ms
      2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,068 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,069 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,155 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms, numFiles: 1
      2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Time: 90ms total, 90.891938ms avg, 90ms max.
      2016-02-01 10:59:21,250 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder - Get block maps: Executed 1 out of 1 using 1 threads. Earliest start: 18.280000 μs, Latest start: 18.280000 μs, Average start: 18.280000 μs .
      2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.fragment.FragmentExecutor - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
      2016-02-01 10:59:21,448 [2950c576-b2d2-5bc3-e9b5-ff4414d088c0:frag:0:0] INFO  o.a.d.e.w.f.FragmentStatusReporter - 2950c576-b2d2-5bc3-e9b5-ff4414d088c0:0:0: State to report: RUNNING
      

      Doing kill -3 PID on the non foreman node for the Drillbit process gives us stack trace in drillbit.out

      2016-02-01 11:03:31
      Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
      
      "qtp801808302-129" #129 prio=5 os_prio=0 tid=0x00007f7ad8127000 nid=0xaad waiting on condition [0x00007f7ab62fb000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
              at java.lang.Thread.run(Thread.java:745)
      
      "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
              - waiting to lock <0x00000007b1283570> (a java.lang.Object)
              at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
              at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-80-acceptor-0-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #80 prio=5 os_prio=0 tid=0x00007f7ad86df000 nid=0x7a7c runnable [0x00007f7ab64fd000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
              - locked <0x00000007b1283570> (a java.lang.Object)
              at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
              at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-79-selector-ServerConnectorManager@6591a1ee/23" #79 prio=5 os_prio=0 tid=0x00007f7ad1368800 nid=0x7a7b runnable [0x00007f7ab65fe000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f70d0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f70c0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f6fa8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-78-selector-ServerConnectorManager@6591a1ee/22" #78 prio=5 os_prio=0 tid=0x00000000026c2800 nid=0x7a7a runnable [0x00007f7ab66ff000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f62e0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f62d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f61b8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-77-selector-ServerConnectorManager@6591a1ee/21" #77 prio=5 os_prio=0 tid=0x00007f7b233e5000 nid=0x7a79 runnable [0x00007f7ab6800000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f54f0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f54e0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f53c8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-76-selector-ServerConnectorManager@6591a1ee/20" #76 prio=5 os_prio=0 tid=0x00007f7ad86dd000 nid=0x7a78 runnable [0x00007f7ab6901000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f4700> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f46f0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f45d8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-74-selector-ServerConnectorManager@6591a1ee/19" #74 prio=5 os_prio=0 tid=0x00007f7ad13cb800 nid=0x7a77 runnable [0x00007f7ab6a01000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f3910> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f3900> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f37e8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-73-selector-ServerConnectorManager@6591a1ee/18" #73 prio=5 os_prio=0 tid=0x00000000026c1000 nid=0x7a76 runnable [0x00007f7ab6b03000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f2b20> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f2b10> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f29f8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-72-selector-ServerConnectorManager@6591a1ee/17" #72 prio=5 os_prio=0 tid=0x00007f7b233e3800 nid=0x7a75 runnable [0x00007f7ab6c04000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13f1ce0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13f1cd0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13f1bb8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-71-selector-ServerConnectorManager@6591a1ee/16" #71 prio=5 os_prio=0 tid=0x00007f7ad86db000 nid=0x7a74 runnable [0x00007f7ab6d05000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b13e14e0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b13e14d0> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b13e13b8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      ...
      
      "Scheduler-897230287" #62 prio=5 os_prio=0 tid=0x00007f7b23302800 nid=0x7a50 waiting on condition [0x00007f7ab770d000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007bb901728> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
              at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "qtp801808302-61-selector-ServerConnectorManager@6591a1ee/7" #61 prio=5 os_prio=0 tid=0x00007f7b233b8000 nid=0x7a4f runnable [0x00007f7ab780e000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b12fb2a0> (a sun.nio.ch.Util$2)
              - locked <0x00000007b12fb290> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b12fb178> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:532)
              at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:485)
              at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      
      ...
      
      "Curator-PathChildrenCache-1" #52 daemon prio=5 os_prio=0 tid=0x00007f7b2308c000 nid=0x7a47 waiting on condition [0x00007f7ab9179000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007bde07d08> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "WorkManager.StatusThread" #40 daemon prio=5 os_prio=0 tid=0x00007f7b229af000 nid=0x7a45 waiting on condition [0x00007f7ab825a000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
              at java.lang.Thread.sleep(Native Method)
              at org.apache.drill.exec.work.WorkManager$StatusThread.run(WorkManager.java:328)
      
      "Curator-PathChildrenCache-0" #51 daemon prio=5 os_prio=0 tid=0x00007f7b2170f800 nid=0x79b2 waiting on condition [0x00007f7ab927a000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b6b98818> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "BitServer-2" #21 daemon prio=10 os_prio=0 tid=0x00007f7b219d3800 nid=0x7981 runnable [0x00007f7ab940d000]
         java.lang.Thread.State: RUNNABLE
              at io.netty.channel.epoll.Native.epollWait0(Native Method)
              at io.netty.channel.epoll.Native.epollWait(Native.java:148)
              at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
              at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)
      
      "UserServer-1" #41 daemon prio=10 os_prio=0 tid=0x00007f7b21846800 nid=0x797f runnable [0x00007f7abb1a6000]
         java.lang.Thread.State: RUNNABLE
              at io.netty.channel.epoll.Native.epollWait0(Native Method)
              at io.netty.channel.epoll.Native.epollWait(Native.java:148)
              at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:180)
              at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:205)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              at java.lang.Thread.run(Thread.java:745)
      
      "Curator-ServiceCache-0" #49 daemon prio=5 os_prio=0 tid=0x00007f7b2184d800 nid=0x796e waiting on condition [0x00007f7abc95d000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b68066c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "Curator-Framework-0" #45 daemon prio=5 os_prio=0 tid=0x00007f7b21064800 nid=0x77ef waiting on condition [0x00007f7aba9a6000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b6806b90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
              at java.util.concurrent.DelayQueue.take(DelayQueue.java:70)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:780)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$300(CuratorFrameworkImpl.java:62)
              at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:257)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      
      "main-EventThread" #44 daemon prio=5 os_prio=0 tid=0x00007f7b21043000 nid=0x77ee waiting on condition [0x00007f7abaaa7000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b6806e88> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491)
      
      "main-SendThread(centos-01.qa.lab:5181)" #43 daemon prio=5 os_prio=0 tid=0x00007f7b20ec9000 nid=0x77ed runnable [0x00007f7abb02d000]
         java.lang.Thread.State: RUNNABLE
              at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
              at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
              at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
              at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
              - locked <0x00000007b5b00220> (a sun.nio.ch.Util$2)
              - locked <0x00000007b5b00230> (a java.util.Collections$UnmodifiableSet)
              - locked <0x00000007b5b001d8> (a sun.nio.ch.EPollSelectorImpl)
              at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
              at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338)
              at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1070)
      
      "Curator-ConnectionStateManager-0" #42 daemon prio=5 os_prio=0 tid=0x00007f7b2223d000 nid=0x77ec waiting on condition [0x00007f7ababa8000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b6807088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
              at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:245)
              at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
              at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:111)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      ...
      
      "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f7b20cf8000 nid=0x7529 in Object.wait() [0x00007f7ac46d5000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
              - locked <0x00000007b6807f58> (a java.lang.ref.ReferenceQueue$Lock)
              at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
              at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
      
      "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f7b20cf0000 nid=0x7528 in Object.wait() [0x00007f7ac47d6000]
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at java.lang.Object.wait(Object.java:502)
              at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
              - locked <0x00000007b5b00d88> (a java.lang.ref.Reference$Lock)
      
      ...
      
      Heap
       garbage-first heap   total 4194304K, used 293887K [0x00000006c0000000, 0x00000006c0204000, 0x00000007c0000000)
        region size 2048K, 145 young (296960K), 21 survivors (43008K)
       Metaspace       used 43409K, capacity 43751K, committed 44284K, reserved 1087488K
        class space    used 5383K, capacity 5510K, committed 5628K, reserved 1048576K
      
      2016-02-01 11:04:08
      Full thread dump OpenJDK 64-Bit Server VM (25.65-b01 mixed mode):
      
      "qtp801808302-130" #130 prio=5 os_prio=0 tid=0x00007f7ad8125000 nid=0xc5b waiting on condition [0x00007f7ab61fa000]
         java.lang.Thread.State: TIMED_WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007b69100a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
              at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:389)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:513)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.access$700(QueuedThreadPool.java:48)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:569)
              at java.lang.Thread.run(Thread.java:745)
      
      "DestroyJavaVM" #84 prio=5 os_prio=0 tid=0x00007f7b20012000 nid=0x74c0 waiting on condition [0x0000000000000000]
         java.lang.Thread.State: RUNNABLE
      
      "qtp801808302-81-acceptor-1-ServerConnector@70c7b8aa{HTTP/1.1}{0.0.0.0:8047}" #81 prio=5 os_prio=0 tid=0x00007f7b233e7000 nid=0x7a7d waiting for monitor entry [0x00007f7ab63fc000]
         java.lang.Thread.State: BLOCKED (on object monitor)
              at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
              - waiting to lock <0x00000007b1283570> (a java.lang.Object)
              at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:336)
              at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:467)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536)
              at java.lang.Thread.run(Thread.java:745)
      

      Attachments

        1. drillbit.out
          97 kB
          Khurram Faraaz

        Issue Links

          Activity

            People

              sudheeshkatkam Sudheesh Katkam
              khfaraaz Khurram Faraaz
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: