[HDDS-1616] ManagedChannel references are being leaked in while removing RaftGroup

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.4.0
• Fix Version/s: 0.5.0
• Component/s: SCM
• Labels: pull-request-available
• Target Version/s: 0.5.0
• Sprint: Done
• Assignee: Mukul Kumar Singh
• Reporter: Mukul Kumar Singh
• Created: 30/May/19 10:13
• Updated: 24/Feb/20 20:52
• Resolved: 02/Jul/19 21:35

Description

ManagedChannel references are being leaked in while removing RaftGroup

May 30, 2019 8:12:20 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=1805, target=192.168.0.3:49867} was not shutdown properly!!! ~*~*~*
    Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
        at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
        at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:411)
        at org.apache.ratis.grpc.client.GrpcClientProtocolClient.<init>(GrpcClientProtocolClient.java:118)
        at org.apache.ratis.grpc.client.GrpcClientRpc.lambda$new$0(GrpcClientRpc.java:55)
        at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.lambda$getProxy$0(PeerProxyMap.java:61)
        at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
        at org.apache.ratis.util.PeerProxyMap$PeerAndProxy.getProxy(PeerProxyMap.java:60)
        at org.apache.ratis.util.PeerProxyMap.getProxy(PeerProxyMap.java:107)
        at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequest(GrpcClientRpc.java:91)
        at org.apache.ratis.client.impl.RaftClientImpl.sendRequest(RaftClientImpl.java:409)
        at org.apache.ratis.client.impl.RaftClientImpl.groupRemove(RaftClientImpl.java:281)
        at org.apache.hadoop.hdds.scm.pipeline.RatisPipelineUtils.destroyPipeline(RatisPipelineUtils.java:97)
        at org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.processPipelineReport(PipelineReportHandler.java:100)
        at org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:80)
        at org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler.onMessage(PipelineReportHandler.java:44)
        at org.apache.hadoop.hdds.server.events.SingleThreadExecutor.lambda$onMessage$1(SingleThreadExecutor.java:85)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #1039