Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-3317

when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged and nothing else is done

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reviewable
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: Future
    • Component/s: Execution - RPC
    • Labels:
      None

      Description

      Trying to reproduce DRILL-3241 I sometimes get the following error in the logs:

      ERROR: Out of memory outside any particular fragment.
      	at org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
      	at org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
      	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
      	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      
      WARN: Failure allocating buffer on incoming stream due to memory limits.  Current Allocation: 1372678764.
      	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
      	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
      	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
      	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
      	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
      	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
      

      ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which calls DataResponseHandlerImpl.informOutOfMemory() which just logs the error in the logs.

      If we have fragments waiting for data they will be stuck waiting forever, and the query will hang (behavior observed in DRILL-3241

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jnadeau Jacques Nadeau
                Reporter:
                adeneche Abdel Hakim Deneche
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: