[HDDS-7929] ECContainerOperationClient should release gRPC client properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Ozone Datanode
• Target Version/s: 1.4.0
• Assignee: Attila Doroszlai
• Reporter: Attila Doroszlai
• Created: 08/Feb/23 17:46
• Updated: 09/Feb/23 09:44
• Resolved: 09/Feb/23 09:44

Description

ECContainerOperationClient releases datanode client with invalidate=false. Upon reconnect this causes warning:

 org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{...} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
	at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
	at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:188)
	at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:158)
	at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:243)
	at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:224)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:224)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:168)
	at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:141)
	at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.<init>(BlockOutputStream.java:161)
	at org.apache.hadoop.hdds.scm.storage.ECBlockOutputStream.<init>(ECBlockOutputStream.java:78)

Issue Links

• duplicates: HDDS-7931 ManagedChannelImpl not cleaned up properly.