[CASSANDRA-4142] OOM Exception during repair session with LeveledCompactionStrategy

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.1.1
• Labels: lcs

Environment

OS: Linux CentOs 6
JDK: Java HotSpot(TM) 64-Bit Server VM (build 14.0-b16, mixed mode)

Node configuration:
Quad-core
10 GB RAM
Xmx set to 2,5 GB (as computed by default).

• Assignee: Sylvain Lebresne
• Reporter: Romain Hardouin

Authors

Sylvain Lebresne

Reviewers

Jonathan Ellis

• Created: 12/Apr/12 07:26
• Updated: 16/Apr/19 09:32
• Resolved: 09/May/12 20:01

Description

We encountered an OOM Exception on 2 nodes during repair session.
Our CF are set up to use LeveledCompactionStrategy and SnappyCompressor.
These two options used together maybe the key to the problem.

Despite of setting XX:+HeapDumpOnOutOfMemoryError, no dump have been generated.
Nonetheless a memory analysis on a live node doing a repair reveals an hotspot: an ArrayList of SSTableBoundedScanner which appears to contain as many objects as there are SSTables on disk.
This ArrayList consumes 786 MB of the heap space for 5757 objects. Therefore each object is about 140 KB.

Eclipse Memory Analyzer's denominator tree shows that 99% of a SSTableBoundedScanner object's memory is consumed by a CompressedRandomAccessReader which contains two big byte arrays.

Cluster information:
9 nodes
Each node handles 35 GB (RandomPartitioner)

This JIRA was created following this discussion:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Why-so-many-SSTables-td7453033.html

Issue Links

• relates to: CASSANDRA-5222 OOM Exception during repair session with LeveledCompactionStrategy