Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.0.0
-
None
-
f7f6efc525cd833ce1530deae32eb9ccb20b664a
Description
I am seeing a memory leak when i Cancel a long running query on sqlline. I am re running the query with assertion enabled, will add details after the second run is complete with assertions.
Long running query was,
select key1, key2 from `twoKeyJsn.json`;
I did Ctrl-C when the above query was running on sqlline, and then issued the below query that returned correct results. And then I see there is a memory leak message in drillbit.log
0: jdbc:drill:schema=dfs.tmp> select count(*) from `twoKeyJsn.json`; +------------+ | EXPR$0 | +------------+ | 26212355 | +------------+ 1 row selected (14.734 seconds)
Stack trace from drillbit.log
2015-05-15 00:59:01,951 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] WARN o.a.drill.exec.ops.SendingAccountor - Interrupted while waiting for send complete. Continuing to wait. java.lang.InterruptedException: null at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) ~[na:1.7.0_45] at java.util.concurrent.Semaphore.acquire(Semaphore.java:472) ~[na:1.7.0_45] at org.apache.drill.exec.ops.SendingAccountor.waitForSendComplete(SendingAccountor.java:48) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContext.waitForSendComplete(FragmentContext.java:436) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.BaseRootExec.close(BaseRootExec.java:112) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.close(ScreenCreator.java:141) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:333) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:278) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from CANCELLATION_REQUESTED --> FAILED for 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from FAILED --> FAILED for 2015-05-15 00:59:01,952 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2aaabb50-3afd-2906-3f48-eb86a315a1f5:0:0: State change requested from FAILED --> FINISHED for 2015-05-15 00:59:01,956 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] ERROR o.a.d.c.exceptions.UserException - SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are zone init allocated delta private 1000000 918080 81920 shared 9999000000 9999000000 0. Fragment 0:0 [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are zone init allocated delta private 1000000 918080 81920 shared 9999000000 9999000000 0. Fragment 0:0 [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:522) ~[drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:315) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:283) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Caused by: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are zone init allocated delta private 1000000 918080 81920 shared 9999000000 9999000000 0. at org.apache.drill.exec.memory.AtomicRemainder.close(AtomicRemainder.java:200) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.memory.Accountor.close(Accountor.java:386) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:325) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:116) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:405) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:394) ~[drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:340) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:278) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] ... 4 common frames omitted 2015-05-15 00:59:01,963 [2aaabb50-3afd-2906-3f48-eb86a315a1f5:frag:0:0] INFO o.a.drill.exec.work.foreman.Foreman - State change requested. CANCELLATION_REQUESTED --> FAILED org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: java.lang.IllegalStateException: Failure while closing accountor. Expected private and shared pools to be set to initial values. However, one or more were not. Stats are zone init allocated delta private 1000000 918080 81920 shared 9999000000 9999000000 0. Fragment 0:0 [Error Id: 90ced8b1-b6db-438f-b193-b7634de31b81 on centos-03.qa.lab:31010] at org.apache.drill.exec.work.foreman.QueryManager$1.statusUpdate(QueryManager.java:481) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryManager$RootStatusReporter.statusChange(QueryManager.java:461) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:90) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.AbstractStatusReporter.fail(AbstractStatusReporter.java:86) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:319) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:283) [drill-java-exec-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.0.0-SNAPSHOT-rebuffed.jar:1.0.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]