Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-1976

Possible Memory Leak in drill jdbc client when dealing with wide columns (5000 chars long)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Execution - Flow
    • Labels:
      None

      Description

      git.commit.id.abbrev=b491cdb

      I am seeing an execution failure when I execute the same query multiple times (<10). The data file contains 9 columns out of which 7 are wide strings (4000-5000 chars long)

      select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on ws.tinyint_var = sub.max_ti
      

      Below are my memory settings :

      DRILL_MAX_DIRECT_MEMORY="32G"
      DRILL_MAX_HEAP="4G"
      

      Error From the JDBC client

      select ws.*, sub.str_var str_var1 from widestrings ws INNER JOIN (select str_var, max(tinyint_var) max_ti from widestrings group by str_var) sub on ws.tinyint_var = sub.max_ti
      
      Exception in pipeline.  Closing channel between local /10.10.100.190:38179 and remote qa-node191.qa.lab/10.10.100.191:31010
      io.netty.handler.codec.DecoderException: java.lang.OutOfMemoryError: Direct buffer memory
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:151)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
      	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
      	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
      	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
      	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
      	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.OutOfMemoryError: Direct buffer memory
      	at java.nio.Bits.reserveMemory(Bits.java:658)
      	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
      	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
      	at io.netty.buffer.PoolArena$DirectArena.newUnpooledChunk(PoolArena.java:443)
      	at io.netty.buffer.PoolArena.allocateHuge(PoolArena.java:187)
      	at io.netty.buffer.PoolArena.allocate(PoolArena.java:165)
      	at io.netty.buffer.PoolArena.reallocate(PoolArena.java:280)
      	at io.netty.buffer.PooledByteBuf.capacity(PooledByteBuf.java:110)
      	at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:251)
      	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:849)
      	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:841)
      	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:831)
      	at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:600)
      	at io.netty.buffer.UnsafeDirectLittleEndian.writeBytes(UnsafeDirectLittleEndian.java:25)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:144)
      	... 13 more
      Channel closed between local /10.10.100.190:38179 and remote qa-node191.qa.lab/10.10.100.191:31010
      Channel is closed, discarding remaining 255231 byte(s) in buffer.
      

      The logs

        Attachments

        1. wide-strings.sh
          7 kB
          Rahul Challapalli
        2. wide-strings-mod.sh
          7 kB
          Rahul Challapalli

          Activity

            People

            • Assignee:
              rkins Rahul Challapalli
              Reporter:
              rkins Rahul Challapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: