[HBASE-22963] Netty ByteBuf leak in rpc client implementation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.3.0, 2.1.7, 2.2.2
• Component/s: rpc
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 01/Sep/19 15:16
• Updated: 26/Jan/22 07:03
• Resolved: 08/Sep/19 14:41

Description

2019-08-31 21:48:27,067 ERROR [RS-EventLoopGroup-6-3] util.ResourceLeakDetector(317): LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	org.apache.hbase.thirdparty.io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:96)
	org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
	org.apache.hbase.thirdparty.io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
	org.apache.hbase.thirdparty.io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
	org.apache.hadoop.hbase.ipc.NettyRpcConnection.<init>(NettyRpcConnection.java:96)
	org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:74)
	org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:40)
	org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:364)
	org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:433)
	org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$300(AbstractRpcClient.java:97)
	org.apache.hadoop.hbase.ipc.AbstractRpcClient$RpcChannelImplementation.callMethod(AbstractRpcClient.java:605)
	org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$Stub.scan(ClientProtos.java:42345)
	org.apache.hadoop.hbase.client.AsyncClientScanner.callOpenScanner(AsyncClientScanner.java:152)
	org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.call(AsyncSingleRequestRpcRetryingCaller.java:82)
	org.apache.hadoop.hbase.client.AsyncSingleRequestRpcRetryingCaller.lambda$doCall$7(AsyncSingleRequestRpcRetryingCaller.java:115)
	org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
	java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	org.apache.hadoop.hbase.client.AsyncRegionLocator.lambda$getRegionLocation$3(AsyncRegionLocator.java:124)
	org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
	java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	org.apache.hadoop.hbase.client.ConnectionUtils.lambda$getOrFetch$6(ConnectionUtils.java:573)
	org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
	java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	org.apache.hadoop.hbase.client.ZKAsyncRegistry.tryComplete(ZKAsyncRegistry.java:123)
	org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getMetaRegionLocation$1(ZKAsyncRegistry.java:165)
	org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
	java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	org.apache.hadoop.hbase.client.ZKAsyncRegistry.lambda$getAndConvert$0(ZKAsyncRegistry.java:81)
	org.apache.hadoop.hbase.util.FutureUtils.lambda$addListener$0(FutureUtils.java:68)
	java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
	java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
	java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
	java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
	org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:174)
	org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:342)
	java.lang.Thread.run(Thread.java:748)

https://builds.apache.org/job/HBase-Flaky-Tests/job/master/4148/testReport/junit/org.apache.hadoop.hbase.client/TestConnection/testConnectionCloseAllowsInterrupt/

Issue Links

• breaks: HBASE-23043 TestWALEntryStream times out
• relates to: HBASE-26708 Netty "leak detected" and OutOfDirectMemoryError due to direct memory buffering with SASL implementation
• links to: GitHub Pull Request #577