Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9681

Memtable heap size grows and many long GC pauses are triggered

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Fixed
    • 2.1.8, 2.2.0 rc2, 3.0 alpha 1
    • None
    • None
    • C* 2.1.7, Debian Wheezy

    • Critical

    Description

      C* 2.1.7 cluster is behaving really bad after 1-2 days. gauges.cassandra.jmx.org.apache.cassandra.metrics.ColumnFamily.AllMemtablesHeapSize.Value jumps to 7 GB (https://www.dropbox.com/s/vraggy292erkzd2/Screenshot%202015-06-29%2019.12.53.png?dl=0) on 3/6 nodes in each data center and then there are many long GC pauses. Cluster is using default heap size values (-Xms8192M -Xmx8192M -Xmn2048M)

      Before C* 2.1.5 memtables heap size was basically constant ~500MB (https://www.dropbox.com/s/fjdywik5lojstvn/Screenshot%202015-06-29%2019.30.00.png?dl=0)

      After restarting all nodes is behaves stable for 1-2days. Today I've done that and long GC pauses are gone (~18:00 https://www.dropbox.com/s/7vo3ynz505rsfq3/Screenshot%202015-06-29%2019.28.37.png?dl=0). The only pattern we've found so far is that long GC pauses are happening basically at the same time on all nodes in the same data center - even on the ones where memtables heap size is not growing.

      Cliffs on the graphs are nodes restarts.

      Used memory on boxes where AllMemtabelesHeapSize grows, stays at the same level - https://www.dropbox.com/s/tes9abykixs86rf/Screenshot%202015-06-29%2019.37.52.png?dl=0.

      Replication factor is set to 3.

      Attachments

        1. cassandra.yaml
          35 kB
          mlowicki
        2. system.log.6.zip
          1.86 MB
          mlowicki
        3. system.log.7.zip
          1.85 MB
          mlowicki
        4. system.log.8.zip
          1.84 MB
          mlowicki
        5. system.log.9.zip
          1.85 MB
          mlowicki
        6. db5.system.log
          3.26 MB
          mlowicki
        7. db5.system.log.1.zip
          853 kB
          mlowicki
        8. db5.system.log.2.zip
          1.24 MB
          mlowicki
        9. db5.system.log.3.zip
          1.74 MB
          mlowicki
        10. schema.cql
          10 kB
          mlowicki

        Activity

          People

            benedict Benedict Elliott Smith
            mlowicki mlowicki
            Benedict Elliott Smith
            T Jake Luciani
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: