[DRILL-5563] Stop non foreman Drillbit results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Component/s: Execution - Flow

Environment

3 node CentOS cluster

• Assignee: Unassigned
• Reporter: Khurram Faraaz
• Created: 05/Jun/17 10:54
• Updated: 06/Jun/17 07:57

Description

Stopping the non-foreman Drillbit normally (as shown below) results in IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.

/opt/mapr/drill/drill-1.11.0/bin/drillbit.sh stop

Drill 1.11.0 commit ID: d11aba2

Details from drillbit.log

Mon Jun  5 09:29:09 UTC 2017 Terminating drillbit pid 28182
2017-06-05 09:29:09,651 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit - Received shutdown request.
2017-06-05 09:29:11,691 [pool-6-thread-1] INFO  o.a.drill.exec.rpc.user.UserServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@55511dc2 in 1004 ms
2017-06-05 09:29:11,691 [pool-6-thread-2] INFO  o.a.drill.exec.rpc.data.DataServer - closed eventLoopGroup io.netty.channel.nio.NioEventLoopGroup@4078d750 in 1004 ms
2017-06-05 09:29:11,692 [pool-6-thread-1] INFO  o.a.drill.exec.service.ServiceEngine - closed userServer in 1005 ms
2017-06-05 09:29:11,692 [pool-6-thread-2] INFO  o.a.drill.exec.service.ServiceEngine - closed dataPool in 1005 ms
2017-06-05 09:29:11,701 [Drillbit-ShutdownHook#0] INFO  o.a.drill.exec.compile.CodeCompiler - Stats: code gen count: 21, cache miss count: 7, hit rate: 67%
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] ERROR o.a.d.exec.server.BootStrapContext - Error while closing
java.lang.IllegalStateException: Allocator[ROOT] closed with outstanding child allocators.
Allocator(ROOT) 0/8000000/201359872/17179869184 (res/actual/peak/limit)
  child allocators: 4
    Allocator(frag:3:2) 2000000/0/0/20000000000 (res/actual/peak/limit)
      child allocators: 0
      ledgers: 0
      reservations: 0
    Allocator(frag:4:2) 2000000/0/0/20000000000 (res/actual/peak/limit)
      child allocators: 0
      ledgers: 0
      reservations: 0
    Allocator(frag:1:2) 2000000/0/0/20000000000 (res/actual/peak/limit)
      child allocators: 0
      ledgers: 0
      reservations: 0
    Allocator(frag:2:2) 2000000/0/0/20000000000 (res/actual/peak/limit)
      child allocators: 0
      ledgers: 0
      reservations: 0
  ledgers: 0
  reservations: 0

        at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:492) ~[drill-memory-base-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.server.BootStrapContext.close(BootStrapContext.java:247) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64) [drill-common-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.server.Drillbit$ShutdownThread.run(Drillbit.java:253) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
2017-06-05 09:29:11,709 [Drillbit-ShutdownHook#0] INFO  o.apache.drill.exec.server.Drillbit - Shutdown completed (2057 ms).