Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9021

AssertionError and Leak detected during sstable compaction

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.1.5
    • None
    • None
    • Normal

    Description

      After ~3 hours of data ingestion we see assertion errors and 'LEAK DETECTED' errors during what looks like sstable compaction.

      system.log snippets (full log attached):

      ...
      INFO  [CompactionExecutor:12] 2015-03-23 02:45:51,770  CompactionTask.java:267 - Compacted 4 sstables to [/mnt/cass_data_disks/data1/requests_ks/timeline-       9500fe40d0f611e495675d5ea01541b5/requests_ks-timeline-ka-185,].  65,916,594 bytes to 66,159,512 (~100% of original) in 26,554ms = 2.376087MB/s.  983 total       partitions merged to 805.  Partition merge counts were {1:627, 2:178, }
      INFO  [CompactionExecutor:11] 2015-03-23 02:45:51,837  CompactionTask.java:267 - Compacted 4 sstables to [/mnt/cass_data_disks/data1/system/                     compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-119,].  426 bytes to 42 (~9% of original) in 82ms = 0.000488MB/s.  5  total partitions merged to 1.  Partition merge counts were {1:1, 2:2, }
      ERROR [NonPeriodicTasks:1] 2015-03-23 02:45:52,251  CassandraDaemon.java:167 - Exception in thread Thread[NonPeriodicTasks:1,5,main]
      java.lang.AssertionError: null
       at org.apache.cassandra.io.compress.CompressionMetadata$Chunk.<init>(CompressionMetadata.java:438) ~[cassandra-all-2.1.3.304.jar:2.1.3.304]
       at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:228) ~[cassandra-all-2.1.3.304.jar:2.1.3.304]
       at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.dropPageCache(CompressedPoolingSegmentedFile.java:80) ~[cassandra-all-2.1.3.304.jar:2.1.3.304]
       at org.apache.cassandra.io.sstable.SSTableReader$6.run(SSTableReader.java:923) ~[cassandra-all-2.1.3.304.jar:2.1.3.304]
       at org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier$1.run(SSTableReader.java:2036) ~[cassandra-all-2.1.3.304.jar:2.1.3.304]
       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_45]
       at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_45]
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) ~[na:1.7.0_45]
       at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) ~[na:1.7.0_45]
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
       at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
      ...
      INFO  [MemtableFlushWriter:50] 2015-03-23 02:47:29,465  Memtable.java:378 - Completed flushing /mnt/cass_data_disks/data1/requests_ks/timeline-                  9500fe40d0f611e495675d5ea01541b5/requests_ks-timeline-ka-188-Data.db (16311981 bytes) for commitlog position ReplayPosition(segmentId=1427071574495,             position=4523631)
      ERROR [Reference-Reaper:1] 2015-03-23 02:47:33,987  Ref.java:181 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@2f59b10) to class org.apache.cassandra.io.sstable.SSTableReader$DescriptorTypeTidy@1251424500:/mnt/cass_data_disks/data1/requests_ks/timeline-9500fe40d0f611e495675d5ea01541b5/    requests_ks-timeline-ka-149 was not released before the reference was garbage collected
      INFO  [Service Thread] 2015-03-23 02:47:40,158  GCInspector.java:142 - ConcurrentMarkSweep GC in 12247ms.  CMS Old Gen: 5318987136 -> 457655168; CMS Perm Gen:   44731264 -> 44699160; Par Eden Space: 8597912 -> 418006664; Par Survivor Space: 71865728 -> 59679584
      ...
      

      Attachments

        1. 9021-improvereporting.txt
          0.6 kB
          Benedict Elliott Smith
        2. 9021.txt
          0.9 kB
          Benedict Elliott Smith
        3. system.log
          2.96 MB
          Rocco Varela

        Activity

          People

            benedict Benedict Elliott Smith
            rocco.varela Rocco Varela
            Benedict Elliott Smith
            Joshua McKenzie
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: