[HDDS-10585] ManagedChannelImpl not closed properly in TestContainerMapper

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: test
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 24/Mar/24 15:24
• Updated: 09/May/24 06:36

Description

Intermittently TestContainerMapper is failing, apparently without any test execution:

[INFO] Running org.apache.hadoop.ozone.fsck.TestContainerMapper
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 24.097 s - in org.apache.hadoop.ozone.fsck.TestContainerMapper

Test output shows some leaked channel:

SEVERE: *~*~*~ Previous channel ManagedChannelImpl{logId=187, target=10.1.0.17:15029} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:102)
	...
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:178)
	...
	at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:237)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:184)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:145)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.<init>(BlockOutputStream.java:168)
	at org.apache.hadoop.hdds.scm.storage.RatisBlockOutputStream.<init>(RatisBlockOutputStream.java:84)
	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.createOutputStream(BlockOutputStreamEntry.java:110)
	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.checkStream(BlockOutputStreamEntry.java:100)
	at org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:136)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.writeToOutputStream(KeyOutputStream.java:250)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:228)
	at org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:208)
	at org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:94)
	at org.apache.hadoop.ozone.fsck.TestContainerMapper.init(TestContainerMapper.java:113)