[HDDS-9351] ManagedChannel not shutdown properly in TestRootedOzoneFileSystemWithFSO

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: test
• Labels: intermittent pull-request-available
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 26/Sep/23 07:34
• Updated: 02/Oct/23 17:12
• Resolved: 30/Sep/23 17:57

Description

Sep 25, 2023 5:59:39 PM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=212, target=10.1.0.144:15031} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:100)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:60)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:51)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:631)
	at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.buildChannel(GrpcClientProtocolClient.java:151)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient.<init>(GrpcClientProtocolClient.java:116)
	at org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:59)
	at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:64)
	at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:270)
	at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:63)
	at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:115)
	at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequestAsync(GrpcClientRpc.java:69)
	at org.apache.ratis.client.impl.OrderedAsync.sendRequest(OrderedAsync.java:245)
	at org.apache.ratis.client.impl.OrderedAsync.sendRequestWithRetry(OrderedAsync.java:198)
	at org.apache.ratis.util.SlidingWindow$Client.sendOrDelayRequest(SlidingWindow.java:312)
	at org.apache.ratis.util.SlidingWindow$Client.submitNewRequest(SlidingWindow.java:291)
	at org.apache.ratis.client.impl.OrderedAsync.send(OrderedAsync.java:170)
	at org.apache.ratis.client.impl.OrderedAsync.newInstance(OrderedAsync.java:118)
	at org.apache.ratis.client.impl.RaftClientImpl.lambda$new$1(RaftClientImpl.java:166)
	at org.apache.ratis.util.MemoizedSupplier.get(MemoizedSupplier.java:62)
	at org.apache.ratis.client.impl.RaftClientImpl.getOrderedAsync(RaftClientImpl.java:235)
	at org.apache.ratis.client.impl.AsyncImpl.send(AsyncImpl.java:41)
	at org.apache.ratis.client.impl.AsyncImpl.send(AsyncImpl.java:46)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.lambda$sendRequestAsync$2(XceiverClientRatis.java:242)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
	at org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendRequestAsync(XceiverClientRatis.java:227)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.sendCommandAsync(XceiverClientRatis.java:329)
	at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.writeChunkAsync(ContainerProtocolCalls.java:420)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunkToContainer(BlockOutputStream.java:722)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.writeChunk(BlockOutputStream.java:539)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlushInternal(BlockOutputStream.java:572)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.handleFlush(BlockOutputStream.java:547)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.close(BlockOutputStream.java:598)
	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.close(BlockOutputStreamEntry.java:158)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleStreamAction(KeyOutputStream.java:523)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleFlushOrClose(KeyOutputStream.java:497)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.close(KeyOutputStream.java:553)
	at org.apache.hadoop.ozone.client.io.OzoneOutputStream.close(OzoneOutputStream.java:103)
	at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:60)
	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:77)
	at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:106)
	at org.apache.hadoop.fs.ozone.TestRootedOzoneFileSystemWithFSO.testDeleteVolumeAndBucket(TestRootedOzoneFileSystemWithFSO.java:221)

• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/21/25574/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.TestRootedOzoneFileSystemWithFSO-output.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/25/25643/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.TestRootedOzoneFileSystemWithFSO-output.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/25/25647/it-filesystem/hadoop-ozone/integration-test/org.apache.hadoop.fs.ozone.TestRootedOzoneFileSystemWithFSO-output.txt

Issue Links

• links to: GitHub Pull Request #5370