Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-7931

ManagedChannelImpl not cleaned up properly.

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      In One of tests, I noticed the following messages:

      Feb 08, 2023 9:06:55 AM org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference cleanQueue
      SEVERE: ~ Channel ManagedChannelImpl{logId=502, target=192.168.1.12:56011} was not shutdown properly!!! ~
          Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
      java.lang.RuntimeException: ManagedChannel allocation site
      at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:93)
      at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
      at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
      at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelImplBuilder.build(ManagedChannelImplBuilder.java:630)
      at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:297)
      at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:188)
      at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:158)
      at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:243)
      at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:224)
      at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
      at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
      at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
      at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
      at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
      at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:224)
      at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:168)
      at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:141)
      at org.apache.hadoop.hdds.scm.storage.BlockOutputStream.<init>(BlockOutputStream.java:161)
      at org.apache.hadoop.hdds.scm.storage.ECBlockOutputStream.<init>(ECBlockOutputStream.java:78)
      at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.getECBlockOutputstream(ECReconstructionCoordinator.java:221)
      at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.reconstructECBlockGroup(ECReconstructionCoordinator.java:278)
      at org.apache.hadoop.ozone.container.ec.reconstruction.ECReconstructionCoordinator.reconstructECContainerGroup(ECReconstructionCoordinator.java:169)
      at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.testECReconstructionCoordinator(TestContainerCommandsEC.java:468)
      at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.lambda$testECReconstructionCoordinatorWithMissingIndexes135$7(TestContainerCommandsEC.java:377)
      at org.junit.Assert.assertThrows(Assert.java:1001)
      at org.junit.Assert.assertThrows(Assert.java:981)
      at org.apache.hadoop.hdds.scm.storage.TestContainerCommandsEC.testECReconstructionCoordinatorWithMissingIndexes135(TestContainerCommandsEC.java:376) 

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            adoroszlai Attila Doroszlai
            ghuangups George Huang
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment