[IGNITE-23396] Java thin: Netty buffer leak on client

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0
• Component/s: thin client
• Labels: ignite-3
• Assignee: Pavel Tupitsyn
• Reporter: Pavel Tupitsyn
• Reviewer: Igor Sapego
• Created: 10/Oct/24 08:27
• Updated: 11/Oct/24 09:31
• Resolved: 11/Oct/24 09:31

Description

Java thin client seems to leak Netty buffers:

 ClientTablesTest > testTableReturnsInstanceWhenExists() STANDARD_ERROR
12:32:26         [2024-10-09T12:32:26,828][ERROR][Test worker][ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
12:32:26         Recent access records:
12:32:26         #1:
12:32:26           org.apache.ignite.internal.client.proto.ClientMessageUnpacker.close(ClientMessageUnpacker.java:884)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.onMessage(TcpClientChannel.java:280)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientConnection.onMessage(NettyClientConnection.java:117)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelRead(NettyClientMessageHandler.java:33)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         #2:
12:32:26           org.apache.ignite.internal.client.proto.ClientMessageUnpacker.retain(ClientMessageUnpacker.java:869)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:477)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.onMessage(TcpClientChannel.java:279)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientConnection.onMessage(NettyClientConnection.java:117)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelRead(NettyClientMessageHandler.java:33)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         #3:
12:32:26           io.netty.buffer.AdvancedLeakAwareByteBuf.readByte(AdvancedLeakAwareByteBuf.java:401)
12:32:26           org.apache.ignite.internal.client.proto.ClientMessageUnpacker.unpackInt(ClientMessageUnpacker.java:105)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:451)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.onMessage(TcpClientChannel.java:279)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientConnection.onMessage(NettyClientConnection.java:117)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelRead(NettyClientMessageHandler.java:33)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         #4:
12:32:26           io.netty.buffer.AdvancedLeakAwareByteBuf.readByte(AdvancedLeakAwareByteBuf.java:401)
12:32:26           org.apache.ignite.internal.client.proto.ClientMessageUnpacker.unpackLong(ClientMessageUnpacker.java:291)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.processNextMessage(TcpClientChannel.java:450)
12:32:26           org.apache.ignite.internal.client.TcpClientChannel.onMessage(TcpClientChannel.java:279)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientConnection.onMessage(NettyClientConnection.java:117)
12:32:26           org.apache.ignite.internal.client.io.netty.NettyClientMessageHandler.channelRead(NettyClientMessageHandler.java:33)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         #5:
12:32:26           Hint: 'NettyClientMessageHandler#0' will handle the message from this point.
12:32:26           io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:115)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:417)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         Created at:
12:32:26           io.netty.buffer.SimpleLeakAwareByteBuf.unwrappedDerived(SimpleLeakAwareByteBuf.java:144)
12:32:26           io.netty.buffer.SimpleLeakAwareByteBuf.retainedSlice(SimpleLeakAwareByteBuf.java:57)
12:32:26           io.netty.buffer.AdvancedLeakAwareByteBuf.retainedSlice(AdvancedLeakAwareByteBuf.java:96)
12:32:26           io.netty.handler.codec.LengthFieldBasedFrameDecoder.extractFrame(LengthFieldBasedFrameDecoder.java:502)
12:32:26           io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:440)
12:32:26           org.apache.ignite.internal.client.proto.ClientMessageDecoder.decode(ClientMessageDecoder.java:56)
12:32:26           io.netty.handler.codec.LengthFieldBasedFrameDecoder.decode(LengthFieldBasedFrameDecoder.java:333)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
12:32:26           io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
12:32:26           io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
12:32:26           io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
12:32:26           io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
12:32:26           io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
12:32:26           io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
12:32:26           io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
12:32:26           io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
12:32:26           io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
12:32:26           io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
12:32:26           java.base/java.lang.Thread.run(Thread.java:834)
12:32:26         : 2 leak records were discarded because the leak record count is targeted to 4. Use system property io.netty.leakDetection.targetRecords to increase the limit.

https://ci.ignite.apache.org/buildConfiguration/ApacheIgnite3xGradle_Test_RunUnitTests_virtual_Batch_3_1/8535527?buildTab=overview&expandBuildDeploymentsSection=false&hideTestsFromDependencies=false&hideProblemsFromDependencies=false&expandBuildChangesSection=true&expandBuildProblemsSection=true

Issue Links

• is related to: IGNITE-23374 Java thin: Netty buffer leak
• links to: GitHub Pull Request #4534