Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10270

Cassandra stops compacting

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Fixed
    • Fix Version/s: 2.2.2, 3.0.0 rc1
    • Component/s: Local/Compaction
    • Labels:
      None
    • Environment:

      linux (google cloud click-to-deploy, default settings, 3 nodes)

    • Severity:
      Critical

      Description

      My cassandra cluster won't keep compacting. I notice that if I restart, it does compact for a while, but after a time it stops. As a result, after adding a bunch of rows, I ended up with about 1000 sstables per node.

      I'll attach more logs in a minute, but it seems like this might be the most relevant part:

      INFO  [CompactionExecutor:1] 2015-09-04 14:22:55,796 CompactionManager.java:1433 - Compaction interrupted: Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2, ranks_by_domain, 812501702/7543091905)bytes
      DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 CompactionManager.java:1437 - Full interruption stack trace:
      org.apache.cassandra.db.compaction.CompactionInterruptedException: Compaction interrupted: Compaction@fff9bcd0-3b1f-11e5-8df6-33158d7bf3bf(megacrawl2, ranks_by_
      domain, 812501702/7543091905)bytes
              at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:180) ~[apache-cassandra-2.2.1.jar:2.2.1]
              at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.1.jar:2.2.1]
              at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.1.jar:2.2.1]
              at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.1.jar:2.2.1]                        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236) ~[apache-cassandra-2.2.1.jar:2.2.1]                                                                                                                                                                       at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.7.0_79]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]                                                                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_79]                                                               at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]                                                                at java.lang.Thread.run(Thread.java:745) [na:1.7.0_79]
      DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 CompactionManager.java:222 - Checking system.local                                                         DEBUG [CompactionExecutor:1] 2015-09-04 14:22:55,797 SizeTieredCompactionStrategy.java:85 - Compaction buckets are [[BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-83-big-Data.db'), BigTableReader(path='/cassandra/data/system/local-7ad54392bcdd35a684174e047860b377/la-81-big-Data.db'),
      

        Attachments

        1. system.log.gz
          2.54 MB
          Adam Bliss
        2. system.txt.gz
          6 kB
          Adam Bliss

          Activity

            People

            • Assignee:
              marcuse Marcus Eriksson
              Reporter:
              abliss Adam Bliss
              Authors:
              Marcus Eriksson
              Reviewers:
              Benedict Elliott Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: