Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-8490

Sender operator fake memory leak result to sql failed and memory statistics error when ChannelClosedException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.21.1
    • 1.22.0
    • Server
    • None

    Description

      1.DES

           **     when ChannelClosedException, .ReconnectingConnection#CloseHandler release sendingAccountor reference counter before netty release buffer, so operator was closed before memory is released by netty .

       

      2 .exception info
       
      2024-04-13 08:45:39,909 [DataClient-3] WARN  o.apache.drill.exec.rpc.RequestIdMap - Failure while attempting to fail rpc response.
      java.lang.IllegalArgumentException: Self-suppression not permitted
              at java.lang.Throwable.addSuppressed(Throwable.java:1072)
              at org.apache.drill.common.DeferredException.addException(DeferredException.java:88)
              at org.apache.drill.common.DeferredException.addThrowable(DeferredException.java:97)
              at org.apache.drill.exec.work.fragment.FragmentExecutor.fail(FragmentExecutor.java:502)
              at org.apache.drill.exec.work.fragment.FragmentExecutor.access$400(FragmentExecutor.java:131)
              at org.apache.drill.exec.work.fragment.FragmentExecutor$ExecutorStateImpl.fail(FragmentExecutor.java:518)
              at org.apache.drill.exec.ops.FragmentContextImpl.fail(FragmentContextImpl.java:298)
              at org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:152)
              at org.apache.drill.exec.ops.FragmentContextImpl$1.accept(FragmentContextImpl.java:149)
              at org.apache.drill.exec.ops.DataTunnelStatusHandler.failed(DataTunnelStatusHandler.java:45)
              at org.apache.drill.exec.rpc.data.DataTunnel$ThrottlingOutcomeListener.failed(DataTunnel.java:125)
              at org.apache.drill.exec.rpc.RequestIdMap$RpcListener.setException(RequestIdMap.java:145)
              at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:78)
              at org.apache.drill.exec.rpc.RequestIdMap$SetExceptionProcedure.apply(RequestIdMap.java:68)
              at com.carrotsearch.hppc.IntObjectHashMap.forEach(IntObjectHashMap.java:692)
              at org.apache.drill.exec.rpc.RequestIdMap.channelClosed(RequestIdMap.java:64)
              at org.apache.drill.exec.rpc.AbstractRemoteConnection.channelClosed(AbstractRemoteConnection.java:192)
              at org.apache.drill.exec.rpc.AbstractClientConnection.channelClosed(AbstractClientConnection.java:97)
              at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:158)
              at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:135)
              at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:205)
              at org.apache.drill.exec.rpc.ReconnectingConnection$CloseHandler.operationComplete(ReconnectingConnection.java:192)
              at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
              at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
              at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
              at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
              at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
              at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
              at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
              at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164)
              at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755)
       
              at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731)
              at io.netty.channel.AbstractChannel$AbstractUnsafe.handleWriteError(AbstractChannel.java:950)
              at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:933)
              at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.forceFlush(AbstractNioChannel.java:361)
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:716)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
              at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
              at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: org.apache.drill.exec.rpc.ChannelClosedException: Channel closed /10.32.112.138:51108 <--> /10.32.112.138:31012.
              at org.apache.drill.exec.rpc.RpcBus$ChannelClosedHandler.operationComplete(RpcBus.java:156)

      *no* further _formatting_ is done here

      Attachments

        Activity

          People

            Unassigned Unassigned
            shihuafeng shihuafeng
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: