[DRILL-3317] when ProtobufLengthDecoder couldn't allocate a new DrillBuf, this error is just logged and nothing else is done

• Type: Bug
• Status: Reviewable
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: Future
• Component/s: Execution - RPC
• Assignee: Jacques Nadeau
• Reporter: Abdel Hakim Deneche
• Created: 18/Jun/15 21:08
• Updated: 26/Aug/20 15:19

Description

Trying to reproduce DRILL-3241 I sometimes get the following error in the logs:

ERROR: Out of memory outside any particular fragment.
	at org.apache.drill.exec.rpc.data.DataResponseHandlerImpl.informOutOfMemory(DataResponseHandlerImpl.java:40)
	at org.apache.drill.exec.rpc.data.DataServer$2.handle(DataServer.java:227)
	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:87)
	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)

WARN: Failure allocating buffer on incoming stream due to memory limits.  Current Allocation: 1372678764.
	at org.apache.drill.exec.rpc.ProtobufLengthDecoder.decode(ProtobufLengthDecoder.java:85)
	at org.apache.drill.exec.rpc.data.DataProtobufLengthDecoder$Server.decode(DataProtobufLengthDecoder.java:52)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:315)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:229)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:324)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:339)

ProtobufLengthDecoder.decode() does call OutOfMemoryHandler.handle() which calls DataResponseHandlerImpl.informOutOfMemory() which just logs the error in the logs.

If we have fragments waiting for data they will be stuck waiting forever, and the query will hang (behavior observed in DRILL-3241

Issue Links

• depends upon: DRILL-3714 Query runs out of memory and remains in CANCELLATION_REQUESTED state until drillbit is restarted
• is related to: DRILL-3241 Query with window function runs out of direct memory and does not report back to client that it did