[CASSANDRA-10449] OOM on bootstrap after long GC pause

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 2.1.x
• Labels: gc

Environment

Ubuntu 14.04, AWS

• Severity: Normal
• Since Version: 2.1.7
• Assignee: Unassigned
• Reporter: Robert Strickland
• Created: 06/Oct/15 00:41
• Updated: 16/Apr/19 09:30
• Resolved: 11/May/16 17:52

Description

I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 500-700GB per node. SSTable counts are <10 per table. I am attempting to provision additional nodes, but bootstrapping OOMs every time after about 10 hours with a sudden long GC pause:

INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
...
ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 CassandraDaemon.java:223 - Exception in thread Thread[MemtableFlushWriter:454,5,main]
java.lang.OutOfMemoryError: Java heap space

I have tried increasing max heap to 48G just to get through the bootstrap, to no avail.

Issue Links

• is duplicated by: CASSANDRA-10680 Deal with small compression chunk size better during streaming plan setup