Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-3594

ManagedChannels are leaked in XceiverClientGrpc manager

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: Ozone Client
    • Labels:
    • Target Version/s:

      Description

      XceiverClientGrpc#ManagedChannels are leaked when running Hadoop Synthetic Load Generator pointing to OzoneFS.

      Stacktrace:

      SEVERE: *~*~*~ Channel ManagedChannelImpl{logId=99, target=10.17.248.31:9859} was not shutdown properly!!! ~*~*~*
          Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
      java.lang.RuntimeException: ManagedChannel allocation site
              at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:94)
              at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:52)
              at org.apache.ratis.thirdparty.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:43)
              at org.apache.ratis.thirdparty.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:518)
              at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connectToDatanode(XceiverClientGrpc.java:191)
              at org.apache.hadoop.hdds.scm.XceiverClientGrpc.connect(XceiverClientGrpc.java:140)
              at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:244)
              at org.apache.hadoop.hdds.scm.XceiverClientManager$2.call(XceiverClientManager.java:228)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3529)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2278)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2155)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache.get(LocalCache.java:3951)
              at org.apache.hadoop.ozone.shaded.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
              at org.apache.hadoop.hdds.scm.XceiverClientManager.getClient(XceiverClientManager.java:228)
              at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:174)
              at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:164)
              at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:184)
              at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:133)
              at org.apache.hadoop.hdds.scm.storage.BlockInputStream.read(BlockInputStream.java:254)
              at org.apache.hadoop.ozone.client.io.KeyInputStream.read(KeyInputStream.java:199)
              at org.apache.hadoop.fs.ozone.OzoneFSInputStream.read(OzoneFSInputStream.java:63)
              at java.io.DataInputStream.read(DataInputStream.java:100)
              at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.read(LoadGenerator.java:284)
              at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.nextOp(LoadGenerator.java:268)
              at org.apache.hadoop.fs.loadGenerator.LoadGenerator$DFSClientThread.run(LoadGenerator.java:235)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                rakeshr Rakesh Radhakrishnan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: