[HDDS-555] RandomKeyGenerator runs not closing the XceiverClient properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.3.0
• Target Version/s: 0.3.0
• Assignee: Mukul Kumar Singh
• Reporter: Shashikant Banerjee
• Created: 26/Sep/18 14:52
• Updated: 12/Oct/18 18:38
• Resolved: 12/Oct/18 17:38

Description

The issue is with closing of the xceiverClients in Ozone. The below shows the leak in XceiverClientGrpc. Similar issue can be seen intermittentkly with XceiverClientRatis as well as shown in https://issues.apache.org/jira/browse/RATIS-318 as well.

Sep 26, 2018 8:11:01 PM org.apache.ratis.shaded.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=136, target=192.168.1.2:50712} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
at org.apache.ratis.shaded.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
at org.apache.ratis.shaded.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:410)
at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:92)
at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:159)
at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:144)
at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4767)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4764)
at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:143)
at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:122)
at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.checkKeyLocationInfo(ChunkGroupOutputStream.java:192)
at org.apache.hadoop.ozone.client.io.ChunkGroupOutputStream.addPreallocateBlocks(ChunkGroupOutputStream.java:180)
at org.apache.hadoop.ozone.client.rpc.RpcClient.createKey(RpcClient.java:472)
at org.apache.hadoop.ozone.client.OzoneBucket.createKey(OzoneBucket.java:262)
at org.apache.hadoop.ozone.freon.RandomKeyGenerator$OfflineProcessor.run(RandomKeyGenerator.java:601)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Issue Links

• duplicates: HDDS-617 XceiverClientManager should cache pipelines on pipelineID