[HDDS-3756] TestBlockOutputStreamWithFailuresFlushDelay leaks channels.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Done
• Labels: unit_test_failure
• Assignee: Unassigned
• Reporter: Mukul Kumar Singh
• Created: 08/Jun/20 15:06
• Updated: 30/May/23 18:23
• Resolved: 30/May/23 18:23

Description

Jun 08, 2020 12:47:25 PM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1708, target=172.17.0.2:38873} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:94)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:52)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:43)
        at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:518)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:194)
        at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:143)
        at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:246)
        at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:230)
        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3951)
        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:230)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:176)
        at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:165)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:184)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:133)
        at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:254)
        at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:199)
        at org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:47)
        at java.io.InputStream.read(InputStream.java:101)
        at org.apache.hadoop.ozone.container.TestHelper.validateData(TestHelper.java:128)
        at org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailuresFlushDelay.validateData(TestBlockOutputStreamWithFailuresFlushDelay.java:857)
        at org.apache.hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailuresFlushDelay.testWatchForCommitDatanodeFailure(TestBlockOutputStreamWithFailuresFlushDelay.java:323)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

Issue Links

• is fixed by: HDDS-7931 ManagedChannelImpl not cleaned up properly.