[HIVE-25706] ShuffleHandler: Make sure of properly releasing netty reference counted objects

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Labels: pull-request-available
• Assignee: László Bodor
• Reporter: László Bodor
• Created: 16/Nov/21 15:53
• Updated: 17/Nov/22 08:47
• Resolved: 05/Feb/22 16:57

Description

https://netty.io/wiki/reference-counted-objects.html

ShuffleHandler log contains issues below when we're on unhappy codepath (sendError). Very easily reproducible e.g. with calling ShuffleHandler's http port without any parameters. Doesn't cause a user-facing issue, because at this point ShuffleHandler already responded to the client, but the exception is annoying and implies invalid .release() calls in netty4.

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
	at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.http.DefaultFullHttpResponse.release(DefaultFullHttpResponse.java:116) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
	at org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.sendError(ShuffleHandler.java:1056) ~[classes/:?]
	at org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.handleRequest(ShuffleHandler.java:827) ~[classes/:?]
	at org.apache.hadoop.hive.llap.shufflehandler.ShuffleHandler$Shuffle.channelRead(ShuffleHandler.java:728) ~[classes/:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-all-4.1.65.Final.jar:4.1.65.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.65.Final.jar:4.1.65.Final]

Issue Links

• relates to: TEZ-4359 ShuffleHandler: Make sure of properly releasing netty reference counted objects
• links to: GitHub Pull Request #2794