Uploaded image for project: 'MINA SSHD'
  1. MINA SSHD
  2. SSHD-1200

ERR - LEAK: ByteBuf.release() was not called before it's garbage-collected.

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.1.0
    • 2.2.0
    • None

    Description

      Expected behavior

      Below error should not come:
      'ERROR ResourceLeakDetector:319 - LEAK: ByteBuf.release() was not called before it's garbage-collected.'

      Actual behavior

      2021-07-21 15:45:00 ERROR ResourceLeakDetector:319 - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
      Recent access records:
      #1:
      io.netty.buffer.AdvancedLeakAwareByteBuf.getBytes(AdvancedLeakAwareByteBuf.java:250)
      org.apache.sshd.netty.NettySupport$1.getRawBytes(NettySupport.java:44)
      org.apache.sshd.common.util.buffer.ByteArrayBuffer.putBuffer(ByteArrayBuffer.java:150)
      org.apache.sshd.common.util.buffer.Buffer.putBuffer(Buffer.java:491)
      org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:510)
      org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:63)
      org.apache.sshd.netty.NettyIoSession.channelRead(NettyIoSession.java:181)
      org.apache.sshd.netty.NettyIoSession$Adapter.channelRead(NettyIoSession.java:216)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
      io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      java.lang.Thread.run(Thread.java:748)
      #2:
      Hint: 'NettyIoSession$Adapter#0' will handle the message from this point.
      io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
      io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      java.lang.Thread.run(Thread.java:748)
      #3:
      Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
      io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
      io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
      io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
      io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      java.lang.Thread.run(Thread.java:748)
      #4:
      io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:634)
      io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
      io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
      io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
      io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      java.lang.Thread.run(Thread.java:748)
      Created at:
      io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
      io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
      io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
      io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:139)
      io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
      io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:150)
      io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
      io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
      io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
      io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
      java.lang.Thread.run(Thread.java:748)

      Steps to reproduce

      NA - The err occurs on a random basis.

      Minimal yet complete reproducer code (or URL to code)

      NA

      Netty version

      1. Initially '4.1.29.Final' version was being used in our app when we noticed the err.
        FYI- We are using the below mentioned jars in our app:
        netty-buffer
        netty-codec
        netty-common
        netty-handler
        netty-resolver
        netty-transport
      1. Then we upgraded above jars to '4.1.65.Final', but still the same error is appearing in our app logs.

      FYI - We are using below apache commons dependencies :-
      commons-codec - 1.10
      commons-io - 2.4
      commons-lang - 2.6
      commons-logging - 1.0.3

      and
      below 'org.apache.sshd' dependencies - version 2.1.0 for all:
      sshd-common
      shd-sftp
      sshd-scp
      sshd-core
      sshd-mina
      sshd-netty
      sshd-putty
      sshd-cli

      JVM version (e.g. java -version)

      Java HotSpot(TM) 64-Bit Server VM (25.281-b25) for linux-amd64 JRE (1.8.0_281-b25)

      OS version (e.g. uname -a)

      Operating System - Linux
      OS Version - 11 SP4

      Is this info sufficient?
      We are checking the exact os version with our Unix team, once we get the details, we will update.

      We had reported this issue to netty, according to them - It's not a Netty bug.
      and they asked us to Release the Bytebuf by ourself (If we've got the option),

      but we are not directly using Bytebuf, it is called by org.apache.sshd.netty.NettySupport$1.getRawBytes(NettySupport.java:44)

      netty mentioned :
      this class is responsible for releasing Bytebuf.

      Just call Bytebuf#release method.
      Refer - https://github.com/netty/netty/issues/11502

      and 
      https://stackoverflow.com/questions/23539854/leak-bytebuf-release-was-not-called-before-its-garbage-collected/23540068

      Could you please fix this issue asap?

      Queries :-

      1. Is this issue fixed in any of the mina-sshd/sshd-netty versions?
        If yes - could you please let us know - which version?
        If not - could you please let us know in which version this fix is expected?
        FYI - I have checked NettySupport.java - getRawBytes() method on 'sshd-2.7.0' version also, but ByteBuf#release is not present in this method.
        https://github.com/apache/mina-sshd/blob/sshd-2.7.0/sshd-netty/src/main/java/org/apache/sshd/netty/NettySupport.java
      2. Could you please check apache commons dependencies which we are using?
        Do we need to also update these libs version to support netty higher version?

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            npdbleak Nilesh Panchal
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment