[DRILL-6441] IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.14.0
• Component/s: Execution - Flow
• Assignee: Unassigned
• Reporter: Khurram Faraaz
• Created: 23/May/18 00:12
• Updated: 25/May/18 08:28

Description

This is seen when the drillbit is shutdown using $DRILL_HOME/bin/drillbit.sh stop, it happens when BaseAllocator tries to close.
Apache Drill 1.14.0 commit c6c5d27d91468a29656bee2acba55d3321978aab

Adding details from Tim's email here

This issue is not related to DRILL-5922, however it is yet another way we leak memory. Looking at the code when you view or submit a query via the rest api as an anonymous user, a session is created with an allocator. If you call drillbit.sh stop during this time, the Drillbit can shutdown while the anonymous session is active and while a child allocator is being used. In this case you will see the error you reported below.

 

 
2018-05-11 15:19:44,510 [2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2509e8fe-f8fb-0212-5bb6-f49d7c611ad0:0:0: State to report: FINISHED
Wed May 16 10:17:13 PDT 2018 Terminating drillbit pid 32076
2018-05-16 10:17:13,793 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Received shutdown request.
2018-05-16 10:17:14,876 [Drillbit-ShutdownHook#0] INFO o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 20, cache miss count: 6, hit rate: 70%
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Allocator(ROOT) 0/0/9577600/34359738368 (res/actual/peak/limit)
 child allocators: 8
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 Allocator(WebServer:AnonUserSession) 0/0/0/9223372036854775807 (res/actual/peak/limit)
 child allocators: 0
 ledgers: 0
 reservations: 0
 ledgers: 0
 reservations: 0

at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:496) ~[drill-memory-base-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:259) ~[drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:81) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:69) [drill-common-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:263) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
 at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:363) [drill-java-exec-1.14.0-SNAPSHOT.jar:1.14.0-SNAPSHOT]
2018-05-16 10:17:14,890 [Drillbit-ShutdownHook#0] INFO o.apache.drill.exec.server.Drillbit - Shutdown completed (1095 ms).