[HDDS-7931] ManagedChannelImpl not cleaned up properly.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Ozone Datanode, SCM Client, test
• Labels: pull-request-available
• Target Version/s: 1.4.0, 1.3.1
• Assignee: Attila Doroszlai
• Reporter: George Huang
• Created: 08/Feb/23 17:52
• Updated: 23/Jun/23 02:44
• Resolved: 16/Feb/23 12:20

Description

In One of tests, I noticed the following messages:

Feb 08, 2023 9:06:55 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: ~ Channel ManagedChannelImpl{logId=502, target=192.168.1.12:56011} was not shutdown properly!!! ~
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:188)
at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:158)
at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:243)
at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:224)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:224)
at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:168)
at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:141)
at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.<init>(BlockOutputStream.java:161)
at org.apache.hadoop.hdds.scm.storage.ECBlockOutputStream.<init>(ECBlockOutputStream.java:78)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.getECBlockOutputstream(ECReconstructionCoordinator.java:221)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.reconstructECBlockGroup(ECReconstructionCoordinator.java:278)
at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.reconstructECContainerGroup(ECReconstructionCoordinator.java:169)
at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.testECReconstructionCoordinator(TestContainerCommandsEC.java:468)
at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.lambda$testECReconstructionCoordinatorWithMissingIndexes135$7(TestContainerCommandsEC.java:377)
at org.junit.Assert.assertThrows(Assert.java:1001)
at org.junit.Assert.assertThrows(Assert.java:981)
at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.testECReconstructionCoordinatorWithMissingIndexes135(TestContainerCommandsEC.java:376) 

Issue Links

• fixes: HDDS-1446 Grpc channels are leaked in XceiverClientGrpc
• fixes: HDDS-1695 XceiverClientGrpc#ManagedChannelReference shutdown issue when running TeraGen
• fixes: HDDS-5017 Grpc channel is not closed when there is exception during read/write
• fixes: HDDS-3594 ManagedChannels are leaked in XceiverClientGrpc manager
• fixes: HDDS-3756 TestBlockOutputStreamWithFailuresFlushDelay leaks channels.
• is duplicated by: HDDS-8797 [hsync] HBase RegionServer input stream not shut down properly
• is duplicated by: HDDS-7929 ECContainerOperationClient should release gRPC client properly
• links to: GitHub Pull Request #4269