Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10515

Commit logs back up with move to 2.1.10



    • Normal


      After upgrading from cassandra 2.0.x to 2.1.10, we began seeing problems where some nodes break the 12G commit log max we configured and go as high as 65G or more before it restarts. Once it reaches the state of more than 12G commit log files, "nodetool compactionstats" hangs. Eventually C* restarts without errors (not sure yet whether it is crashing but I'm checking into it) and the cleanup occurs and the commit logs shrink back down again. Here is the nodetool compactionstats immediately after restart.

      jgriffith@prod1xc1.c2.bf1:~$ ndc
      pending tasks: 2185
         compaction type   keyspace                          table     completed          total    unit   progress
              Compaction   SyncCore                          *cf1*   61251208033   170643574558   bytes     35.89%
              Compaction   SyncCore                          *cf2*   19262483904    19266079916   bytes     99.98%
              Compaction   SyncCore                          *cf3*    6592197093     6592316682   bytes    100.00%
              Compaction   SyncCore                          *cf4*    3411039555     3411039557   bytes    100.00%
              Compaction   SyncCore                          *cf5*    2879241009     2879487621   bytes     99.99%
              Compaction   SyncCore                          *cf6*   21252493623    21252635196   bytes    100.00%
              Compaction   SyncCore                          *cf7*   81009853587    81009854438   bytes    100.00%
              Compaction   SyncCore                          *cf8*    3005734580     3005768582   bytes    100.00%
      Active compaction remaining time :        n/a

      I was also doing periodic "nodetool tpstats" which were working but not being logged in system.log on the StatusLogger thread until after the compaction started working again.


        1. C5commitLogIncrease.jpg
          123 kB
          Jeff Griffith
        2. cassandra.yaml
          3 kB
          Jeff Griffith
        3. CASSANDRA-19579.jpg
          97 kB
          Jeff Griffith
        4. cfstats-clean.txt
          110 kB
          Jeff Griffith
        5. CommitLogProblem.jpg
          351 kB
          Jeff Griffith
        6. CommitLogSize.jpg
          60 kB
          Jeff Griffith
        7. MultinodeCommitLogGrowth-node1.tar.gz
          18.62 MB
          Jeff Griffith
        8. RUN3tpstats.jpg
          454 kB
          Jeff Griffith
        9. stacktrace.txt
          2.00 MB
          Jeff Griffith
        10. system.log.clean
          2.63 MB
          Jeff Griffith



            blambov Branimir Lambov
            jeffery.griffith Jeff Griffith
            Branimir Lambov
            0 Vote for this issue
            9 Start watching this issue