[HDDS-1695] XceiverClientGrpc#ManagedChannelReference shutdown issue when running TeraGen

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Works for Me
• Assignee: Unassigned
• Reporter: Xiaoyu Yao
• Created: 17/Jun/19 15:48
• Updated: 21/Jan/24 15:48
• Resolved: 21/Jan/24 15:48

Description

Jun 17, 2019 8:45:22 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl\{logId=1, target=10.17.207.52:9859} was not shutdown properly!!! ~*~*~*
 Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
 at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
 at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
 at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
 at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:172)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.reconnect(XceiverClientGrpc.java:417)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandAsync(XceiverClientGrpc.java:366)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:279)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:248)
 at org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:231)
 at org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:118)
 at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:167)
 at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:118)
 at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:222)
 at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:171)
 at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:52)
 at java.io.DataInputStream.read(DataInputStream.java:149)
 at org.apache.hadoop.examples.terasort.TeraInputFormat$TeraRecordReader.nextKeyValue(TeraInputFormat.java:257)
 at org.apache.hadoop.examples.terasort.TeraInputFormat$1.run(TeraInputFormat.java:154)

Issue Links

• is fixed by: HDDS-7931 ManagedChannelImpl not cleaned up properly.
• is related to: HDDS-5017 Grpc channel is not closed when there is exception during read/write
• relates to: HDDS-1446 Grpc channels are leaked in XceiverClientGrpc
• relates to: HDDS-3600 ManagedChannels leaked on ratis pipeline when there are many connection retries