Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6496

Endless L0 LCS compactions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.4
    • None
    • Cassandra 2.0.3, Linux, 6 nodes, 5 disks per node

    Description

      I have first described the problem here: http://stackoverflow.com/questions/20589324/cassandra-2-0-3-endless-compactions-with-no-traffic

      I think I have really abused my system with the traffic (mix of reads, heavy updates and some deletes). Now after stopping the traffic I see the compactions that are going on endlessly for over 4 days.

      For a specific CF I have about 4700 sstable data files right now. The compaction estimates are logged as "[3312, 4, 0, 0, 0, 0, 0, 0, 0]". sstable_size_in_mb=256. 3214 files are about 256Mb (+/1 few megs), other files are smaller or much smaller than that. No sstables are larger than 256Mb. What I observe is that LCS picks 32 sstables from L0 and compacts them into 32 sstables of approximately the same size. So, what my system is doing for last 4 days (no traffic at all) is compacting groups of 32 sstables into groups of 32 sstables without any changes. Seems like a bug to me regardless of what did I do to get the system into this state...

      Attachments

        1. 6496.txt
          12 kB
          Jonathan Ellis
        2. system.log.gz
          112 kB
          Nikolai Grigoriev
        3. system.log.1.gz
          1.26 MB
          Nikolai Grigoriev

        Activity

          People

            jbellis Jonathan Ellis
            ngrigoriev Nikolai Grigoriev
            Jonathan Ellis
            Marcus Eriksson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: