Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5718

java.lang.IllegalStateException: Memory was leaked by query

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.9.0, 1.11.0
    • Fix Version/s: None
    • Labels:
      None
    • Environment:

      Description

      Configurations

      planner.memory.max_query_memory_per_node: 17179869184 (16 GB)
      planner.width.max_per_node: 48
      store.parquet.block-size: 134217728 (128 MB, this is the block size used to create the parquet files)
      
      Fragment 0:0
      
      [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
      [BitServer-4] INFO org.apache.drill.exec.work.fragment.FragmentExecutor - 267104f2-e48d-1d66-63f4-387848c1ccf2:1:10: State change requested RUNNING --> CANCELLATION_REQUESTED
      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: ChannelClosedException: Channel closed /127.0.0.1:31010 <--> /127.0.0.1:40404.
      
      Fragment 0:0
      
      [Error Id: 05c39a1e-c8a8-4147-870f-e0cdbb454e53 on iWebStitchFixDev:31010]
              at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
              at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:295)
              at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
              at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
              at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed /127.0.0.1:31010 <--> /127.0.0.1:40404.
              at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:164)
              at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:144)
              at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
              at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
              at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
              at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
              at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
              at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:943)
              at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:592)
              at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:584)
              at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1099)
              at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
              at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
              at io.netty.channel.ChannelOutboundHandlerAdapter.close(ChannelOutboundHandlerAdapter.java:71)
              at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:615)
              at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:600)
              at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:466)
              at io.netty.handler.timeout.ReadTimeoutHandler.readTimedOut(ReadTimeoutHandler.java:187)
              at org.apache.drill.exec.rpc.BasicServer$LoggingReadTimeoutHandler.readTimedOut(BasicServer.java:122)
              at io.netty.handler.timeout.ReadTimeoutHandler$ReadTimeoutTask.run(ReadTimeoutHandler.java:212)
              at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
              at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120)
              at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
              ... 1 more
      Suppressed: org.apache.drill.exec.rpc.RpcException: Failure sending message.
                      at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:124)
                      at org.apache.drill.exec.rpc.user.UserServer$BitToUserConnection.sendData(UserServer.java:173)
                      at org.apache.drill.exec.ops.AccountingUserConnection.sendData(AccountingUserConnection.java:42)
                      at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:118)
                      at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:95)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:234)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:227)
                      at java.security.AccessController.doPrivileged(Native Method)
                      at javax.security.auth.Subject.doAs(Subject.java:422)
                      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:227)
                      at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                      ... 1 more
              Caused by: java.lang.IllegalArgumentException: Attempted to send a message when connection is no longer valid.
                      at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
                      at org.apache.drill.exec.rpc.RequestIdMap.createNewRpcListener(RequestIdMap.java:88)
                      at org.apache.drill.exec.rpc.AbstractRemoteConnection.createNewRpcListener(AbstractRemoteConnection.java:162)
                      at org.apache.drill.exec.rpc.RpcBus.send(RpcBus.java:117)
                      ... 14 more
              Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (5259264)
      Allocator(op:0:0:0:Screen) 1000000/5259264/11538432/10000000000 (res/actual/peak/limit)
      
                      at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:521)
                      at org.apache.drill.exec.ops.AbstractOperatorExecContext.close(AbstractOperatorExecContext.java:86)
                      at org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:108)
                      at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:436)
                      at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:425)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:320)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
                      at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                      ... 1 more
              Suppressed: org.apache.drill.exec.rpc.ChannelClosedException: java.nio.channels.ClosedChannelException
                      at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.operationComplete(RequestIdMap.java:122)
                      at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.operationComplete(RequestIdMap.java:98)
                      at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
                      at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
                      at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
                      at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
                      at io.netty.channel.ChannelOutboundBuffer.safeFail(ChannelOutboundBuffer.java:645)
                      at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:290)
                      at io.netty.channel.ChannelOutboundBuffer.failFlushed(ChannelOutboundBuffer.java:589)
                      at io.netty.channel.AbstractChannel$AbstractUnsafe.closeAndDeregister(AbstractChannel.java:603)
                      at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:585)
                      ... 16 more
              Caused by: java.nio.channels.ClosedChannelException
              Suppressed: java.lang.IllegalStateException: Memory was leaked by query. Memory leaked: (1000000)
      Allocator(frag:0:0) 5000000/1000000/1146165504/50000000000 (res/actual/peak/limit)
      
                      at org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:521)
                      at org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:436)
                      at org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:430)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:320)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
                      at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:264)
                      at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
                      ... 1 more
      

        Attachments

        1. drillbit.out.tar.gz
          29 kB
          Mohamed Mohsen

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mgelbana Mohamed Mohsen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: