[HDDS-3594] ManagedChannels are leaked in XceiverClientGrpc manager

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.0.0
• Component/s: Ozone Client
• Labels: TriagePending
• Target Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Rakesh Radhakrishnan
• Created: 15/May/20 09:56
• Updated: 13/Feb/23 04:59
• Resolved: 10/Jul/20 11:11

Description

XceiverClientGrpc#ManagedChannels are leaked when running Hadoop Synthetic Load Generator pointing to OzoneFS.

Stacktrace:

SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=99, target=10.17.248.31:9859} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:94)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:52)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:43)
        at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:518)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:191)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:140)
        at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:244)
        at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:228)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
        at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:228)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:174)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:164)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:184)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:133)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:254)
        at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:199)
        at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:63)
        at java.io.DataInputStream.read(DataInputStream.java:100)
        at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.read(LoadGenerator.java:284)
        at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.nextOp(LoadGenerator.java:268)
        at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.run(LoadGenerator.java:235)

Issue Links

• is duplicated by: HDDS-3600 ManagedChannels leaked on ratis pipeline when there are many connection retries
• is related to: HDDS-7931 ManagedChannelImpl not cleaned up properly.