[CASSANDRA-9681] Memtable heap size grows and many long GC pauses are triggered

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1.8, 2.2.0 rc2, 3.0 alpha 1

Environment

C* 2.1.7, Debian Wheezy

• Severity: Critical
• Assignee: Benedict Elliott Smith Assign to me
• Reporter: mlowicki

Authors

Benedict Elliott Smith

Reviewers

T Jake Luciani

• Created: 29/Jun/15 17:32
• Updated: 16/Apr/19 09:31
• Resolved: 02/Jul/15 09:37

Description

C* 2.1.7 cluster is behaving really bad after 1-2 days. gauges.cassandra.jmx.org.apache.cassandra.metrics.ColumnFamily.AllMemtablesHeapSize.Value jumps to 7 GB (https://www.dropbox.com/s/vraggy292erkzd2/Screenshot%202015-06-29%2019.12.53.png?dl=0) on 3/6 nodes in each data center and then there are many long GC pauses. Cluster is using default heap size values (-Xms8192M -Xmx8192M -Xmn2048M)

Before C* 2.1.5 memtables heap size was basically constant ~500MB (https://www.dropbox.com/s/fjdywik5lojstvn/Screenshot%202015-06-29%2019.30.00.png?dl=0)

After restarting all nodes is behaves stable for 1-2days. Today I've done that and long GC pauses are gone (~18:00 https://www.dropbox.com/s/7vo3ynz505rsfq3/Screenshot%202015-06-29%2019.28.37.png?dl=0). The only pattern we've found so far is that long GC pauses are happening basically at the same time on all nodes in the same data center - even on the ones where memtables heap size is not growing.

Cliffs on the graphs are nodes restarts.

Used memory on boxes where AllMemtabelesHeapSize grows, stays at the same level - https://www.dropbox.com/s/tes9abykixs86rf/Screenshot%202015-06-29%2019.37.52.png?dl=0.

Replication factor is set to 3.