Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7640

Seeing commit log grow unbounded

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Low
    • Resolution: Duplicate
    • Fix Version/s: 2.0.10
    • Component/s: None
    • Labels:
      None
    • Severity:
      Low

      Description

      While running 2.0.6 saw the commit log size grow up to 30 GB before restarting the node, with the max set to 1 GB.

      With DEBUG logging on CommitLog* we see segments are not being removed because they are still dirty. But for some reason the force flush when the active size is too large isn't happening (The active size gets printed out, and is 20+GB). The logs show normal "high volume" flushes happening, and force flushes of system.* stuff happening. But there are no flushes occurring for the memtables that are keeping the segments dirty.

      Upon restart the commit log replays fine with no errors. There are no errors from flushing to the post flusher. tpstats for both show 0 active 0 pending at times during.

      The only config change around this is that the segment size is larger than normal, it is set to 128 MB, as there are some very large columns being inserted (30+MB).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                benedict Benedict Elliott Smith
                Reporter:
                jjordan Jeremiah Jordan
                Authors:
                Benedict Elliott Smith
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: