Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16243

LEAK DETECTED: was not released before the reference was garbage collected

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Local/Compaction
    • Labels:
      None
    • Bug Category:
      Degradation - Resource Management
    • Severity:
      Critical
    • Complexity:
      Challenging
    • Discovered By:
      User Report
    • Platform:
      All
    • Impacts:
      None

      Description

      Hello,

      I am getting the following errors and I am running 3.11.5

      LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910) was not released before the reference was garbage collected
      
      
      ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3763,1,main]
      org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
              at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
              at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
              at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
      Caused by: java.io.IOException: Map failed
              at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
              at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
              ... 23 common frames omitted
      Caused by: java.lang.OutOfMemoryError: Map failed
              at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
              at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
              ... 24 common frames omitted
      WARN  [GossipTasks:1] 2020-11-02 02:34:53,302  FailureDetector.java:278 - Not marking nodes down due to local pause of 7468254535 > 5000000000
      ERROR [Reference-Reaper] 2020-11-02 02:34:53,208  Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db was not released before the reference was garbage collected
      ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3762,1,main]
      org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
              at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
              at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
              at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
      Caused by: java.io.IOException: Map failed
              at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
              at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
              ... 23 common frames omitted
      Caused by: java.lang.OutOfMemoryError: Map failed
              at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
              at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
              ... 24 common frames omitted
      INFO  [Service Thread] 2020-11-02 02:34:53,877  StatusLogger.java:51 - MemtablePostFlush                 0         0           3596         0                 0 

       

      These are the settings

      # /etc/security/limits-cassandra.conf
      # Cassandra-specfic parameters
      # add
      cassandra        soft    memlock         unlimited
      cassandra        hard    memlock         unlimited
      cassandra        soft    nproc           65535
      #change
      *                soft    nofile          655350
      *                hard    nofile          655350 

      For cassandra process

      cat /proc/25586/limits
      Limit                     Soft Limit           Hard Limit           Units
      Max cpu time              unlimited            unlimited            seconds
      Max file size             unlimited            unlimited            bytes
      Max data size             unlimited            unlimited            bytes
      Max stack size            8388608              unlimited            bytes
      Max core file size        0                    0                    bytes
      Max resident set          unlimited            unlimited            bytes
      Max processes             65535                692334               processes
      Max open files            655350               655350               files
      Max locked memory         unlimited            unlimited            bytes
      Max address space         unlimited            unlimited            bytes
      Max file locks            unlimited            unlimited            locks
      Max pending signals       692334               692334               signals
      Max msgqueue size         819200               819200               bytes
      Max nice priority         0                    0
      Max realtime priority     0                    0
      Max realtime timeout      unlimited            unlimited            us 

      vm.max_map_count is set to 1048575

      sysctl vm.max_map_count
      vm.max_map_count = 1048575

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                prabin.sh@gmail.com developthou
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: