Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-7120

Query fails with ChannelClosedException when Statistics is disabled

    XMLWordPrintableJSON

    Details

      Description

      TPCH query 5 fails at sf100 when Statistics is disabled. Here is the query:

      select
        n.n_name,
        sum(l.l_extendedprice * (1 - l.l_discount)) as revenue
      from
        customer c,
        orders o,
        lineitem l,
        supplier s,
        nation n,
        region r
      where
        c.c_custkey = o.o_custkey
        and l.l_orderkey = o.o_orderkey
        and l.l_suppkey = s.s_suppkey
        and c.c_nationkey = s.s_nationkey
        and s.s_nationkey = n.n_nationkey
        and n.n_regionkey = r.r_regionkey
        and r.r_name = 'EUROPE'
        and o.o_orderdate >= date '1997-01-01'
        and o.o_orderdate < date '1997-01-01' + interval '1' year
      group by
        n.n_name
      order by
        revenue desc;
      

      This is the error from drillbit.log:

      2019-03-04 17:46:38,684 [23822b0a-b7bd-0b79-b905-1438f5b1d039:frag:6:64] INFO  o.a.d.e.w.fragment.FragmentExecutor - 23822b0a-b7bd-0b79-b905-1438f5b1d039:6:64: State change requested RUNNING --> FINISHED
      2019-03-04 17:46:38,684 [23822b0a-b7bd-0b79-b905-1438f5b1d039:frag:6:64] INFO  o.a.d.e.w.f.FragmentStatusReporter - 23822b0a-b7bd-0b79-b905-1438f5b1d039:6:64: State to report: FINISHED
      2019-03-04 18:17:51,454 [BitServer-13] WARN  o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits.  Current Allocation: 262144.
      2019-03-04 18:17:51,454 [BitServer-13] ERROR o.a.drill.exec.rpc.data.DataServer - Out of memory in RPC layer.
      2019-03-04 18:17:51,463 [BitServer-13] ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  Connection: /10.10.120.104:31012 <--> /10.10.120.106:53048 (data server).  Closing connection.
      io.netty.handler.codec.DecoderException: org.apache.drill.exec.exception.OutOfMemoryException: Failure allocating buffer.
              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:271) ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-common-4.0.48.Final.jar:4.0.48.Final]
              at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
      Caused by: org.apache.drill.exec.exception.OutOfMemoryException: Failure allocating buffer.
              at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:67) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              at org.apache.drill.exec.memory.AllocationManager.<init>(AllocationManager.java:84) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
              at org.apache.drill.exec.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:258) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
              at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:241) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
              at org.apache.drill.exec.memory.BaseAllocator.buffer(BaseAllocator.java:211) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:1.16.0-SNAPSHOT]
              at io.netty.buffer.ExpandableByteBuf.capacity(ExpandableByteBuf.java:43) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:270) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:867) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:859) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.handler.codec.ByteToMessageDecoder$1.cumulate(ByteToMessageDecoder.java:94) ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.0.48.Final.jar:4.0.48.Final]
              ... 18 common frames omitted
      Caused by: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 107374182400, max: 107374182400)
              at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:510) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:464) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:766) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:742) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-buffer-4.0.48.Final.jar:4.0.48.Final]
              at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:169) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:169) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:201) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:65) ~[drill-memory-base-1.16.0-SNAPSHOT.jar:4.0.48.Final]
              ... 30 common frames omitted
      2019-03-04 18:17:51,464 [BitServer-13] INFO  o.a.d.exec.rpc.ProtobufLengthDecoder - Channel is closed, discarding remaining 95971 byte(s) in buffer.
      2019-03-04 18:17:51,472 [BitServer-13] WARN  o.a.d.exec.rpc.ProtobufLengthDecoder - Failure allocating buffer on incoming stream due to memory limits.  Current Allocation: 335872.
      

      This query completes when Statistics is enabled.

      I am using commit 469be17597e7b7c6bc1de9863dcb6c5604a55f0c.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                gparai Gautam Parai
                Reporter:
                rhou Robert Hou
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: