Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7373

Commit logs no longer deleting and MemtablePostFlusher pending growing

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 1.2.17, 2.0.9, 2.1 rc2
    • Component/s: None
    • Labels:
      None
    • Environment:

      RHEL 6.5
      Cassandra 1.12.16
      Replication factor of 3

    • Severity:
      Normal

      Description

      We have this issue where once in a while, we get into a situation where the MemtablePostFlusher is not executing and the space used by the commit logs on disks keeps on increasing and increasing.

      We can observe the problem by invoking nodetool tpstats:

      Pool Name                    Active   Pending      Completed   Blocked  All time blocked
      ReadStage                         6         6       46650213         0                 0
      RequestResponseStage              0         0      130547421         0                 0
      MutationStage                     2         2      116813206         0                 0
      ReadRepairStage                   0         0        2322201         0                 0
      ReplicateOnWriteStage             0         0              0         0                 0
      GossipStage                       0         0         120780         0                 0
      AntiEntropyStage                  0         0              0         0                 0
      MigrationStage                    0         0              0         0                 0
      MemoryMeter                       0         0            456         0                 0
      MemtablePostFlusher               1       447           6344         0                 0
      FlushWriter                       0         0           6132         0                62
      MiscStage                         0         0              0         0                 0
      PendingRangeCalculator            0         0              6         0                 0
      commitlog_archiver                0         0              0         0                 0
      InternalResponseStage             0         0              0         0                 0
      HintedHandoff                     2         2              4         0                 0
      
      Message type           Dropped
      RANGE_SLICE                  0
      READ_REPAIR                  0
      BINARY                       0
      READ                         0
      MUTATION                     0
      _TRACE                       0
      REQUEST_RESPONSE             0
      COUNTER_MUTATION             0
      

      Here is a potential error in the logs that can explain this:

      ERROR [FlushWriter:2693] 2014-06-09 22:05:38,452 CassandraDaemon.java (line 191) Exception in thread Thread[FlushWriter:2693,5,main]
      java.lang.NegativeArraySizeException
      	at org.apache.cassandra.io.util.FastByteArrayOutputStream.expand(FastByteArrayOutputStream.java:104)
      	at org.apache.cassandra.io.util.FastByteArrayOutputStream.write(FastByteArrayOutputStream.java:220)
      	at java.io.DataOutputStream.write(DataOutputStream.java:107)
      	at org.apache.cassandra.io.util.DataOutputBuffer.write(DataOutputBuffer.java:60)
      	at org.apache.cassandra.utils.ByteBufferUtil.write(ByteBufferUtil.java:328)
      	at org.apache.cassandra.utils.ByteBufferUtil.writeWithLength(ByteBufferUtil.java:315)
      	at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:55)
      	at org.apache.cassandra.db.ColumnSerializer.serialize(ColumnSerializer.java:30)
      	at org.apache.cassandra.db.OnDiskAtom$Serializer.serializeForSSTable(OnDiskAtom.java:62)
      	at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:181)
      	at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:133)
      	at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:185)
      	at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:430)
      	at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:385)
      	at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
      
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              mishail Mikhail Stepura Assign to me
              Reporter:
              frichard Francois Richard
              Authors:
              Mikhail Stepura
              Reviewers:
              Jonathan Ellis

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment