[SSHD-1200] ERR - LEAK: ByteBuf.release() was not called before it's garbage-collected.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.1.0
• Fix Version/s: 2.2.0
• External issue URL: https://github.com/netty/netty/issues/11502
• Assignee: Unassigned
• Reporter: Nilesh Panchal
• Created: 23/Jul/21 06:25
• Updated: 23/Jul/21 08:43
• Resolved: 23/Jul/21 08:43

Description

Expected behavior

Below error should not come:
'ERROR ResourceLeakDetector:319 - LEAK: ByteBuf.release() was not called before it's garbage-collected.'

Actual behavior

2021-07-21 15:45:00 ERROR ResourceLeakDetector:319 - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
#1:
io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:250)
org.apache.sshd.netty.NettySupport$1.getRawBytes(NettySupport.java:44)
org.apache.sshd.common.util.buffer.ByteArrayBuffer.putBuffer(ByteArrayBuffer.java:150)
org.apache.sshd.common.util.buffer.Buffer.putBuffer(Buffer.java:491)
org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:510)
org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63)
org.apache.sshd.netty.NettyIoSession.channelRead(NettyIoSession.java:181)
org.apache.sshd.netty.NettyIoSession$Adapter.channelRead(NettyIoSession.java:216)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#2:
Hint: 'NettyIoSession$Adapter#0' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#3:
Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
#4:
io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:748)

Steps to reproduce

NA - The err occurs on a random basis.

Minimal yet complete reproducer code (or URL to code)

NA

Netty version

1. Initially '4.1.29.Final' version was being used in our app when we noticed the err.
   FYI- We are using the below mentioned jars in our app:
   netty-buffer
   netty-codec
   netty-common
   netty-handler
   netty-resolver
   netty-transport

1. Then we upgraded above jars to '4.1.65.Final', but still the same error is appearing in our app logs.

FYI - We are using below apache commons dependencies :-
commons-codec - 1.10
commons-io - 2.4
commons-lang - 2.6
commons-logging - 1.0.3

and
below 'org.apache.sshd' dependencies - version 2.1.0 for all:
sshd-common
shd-sftp
sshd-scp
sshd-core
sshd-mina
sshd-netty
sshd-putty
sshd-cli

JVM version (e.g. java -version)

Java HotSpot(TM) 64-Bit Server VM (25.281-b25) for linux-amd64 JRE (1.8.0_281-b25)

OS version (e.g. uname -a)

Operating System - Linux
OS Version - 11 SP4

Is this info sufficient?
We are checking the exact os version with our Unix team, once we get the details, we will update.

We had reported this issue to netty, according to them - It's not a Netty bug.
and they asked us to Release the Bytebuf by ourself (If we've got the option),

but we are not directly using Bytebuf, it is called by org.apache.sshd.netty.NettySupport$1.getRawBytes(NettySupport.java:44)

netty mentioned :
this class is responsible for releasing Bytebuf.

Just call Bytebuf#release method.
Refer - https://github.com/netty/netty/issues/11502

and 
https://stackoverflow.com/questions/23539854/leak-bytebuf-release-was-not-called-before-its-garbage-collected/23540068

Could you please fix this issue asap?

Queries :-

1. Is this issue fixed in any of the mina-sshd/sshd-netty versions?
   If yes - could you please let us know - which version?
   If not - could you please let us know in which version this fix is expected?
   FYI - I have checked NettySupport.java - getRawBytes() method on 'sshd-2.7.0' version also, but ByteBuf#release is not present in this method.
   https://github.com/apache/mina-sshd/blob/sshd-2.7.0/sshd-netty/src/main/java/org/apache/sshd/netty/NettySupport.java
2. Could you please check apache commons dependencies which we are using?
   Do we need to also update these libs version to support netty higher version?

Issue Links

• duplicates: SSHD-833 Netty: ByteBuf LEAK message