[HDDS-1446] Grpc channels are leaked in XceiverClientGrpc

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Works for Me
• Affects Version/s: 0.3.0
• Component/s: Ozone Client
• Assignee: Attila Doroszlai
• Reporter: Mukul Kumar Singh
• Created: 17/Apr/19 07:09
• Updated: 14/Dec/23 11:55
• Resolved: 14/Dec/23 11:55

Description

Grpc Channels are leaked in MiniOzoneChaosCluster runs.

SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=522, target=10.200.4.160:52415} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
        at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:165)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:389)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:340)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:268)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:236)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:210)
        at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:119)
        at org.apache.hadoop.ozone.client.io.KeyInputStream.getFromOmKeyInfo(KeyInputStream.java:302)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.createInputStream(RpcClient.java:993)
        at org.apache.hadoop.ozone.client.rpc.RpcClient.getKey(RpcClient.java:653)
        at org.apache.hadoop.ozone.client.OzoneBucket.readKey(OzoneBucket.java:325)
        at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:112)
        at org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:147)
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Issue Links

• Is contained by: HDDS-1525 Mapreduce failure when using Hadoop 2.7.5
• is fixed by: HDDS-7931 ManagedChannelImpl not cleaned up properly.
• is related to: HDDS-1695 XceiverClientGrpc#ManagedChannelReference shutdown issue when running TeraGen
• is related to: HDDS-3600 ManagedChannels leaked on ratis pipeline when there are many connection retries
• is related to: HDDS-5017 Grpc channel is not closed when there is exception during read/write