[CASSANDRA-11383] Avoid index segment stitching in RAM which lead to OOM on big SSTable files

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.5
• Component/s: Legacy/CQL
• Labels: sasi

Environment

C* 3.4

• Severity: Normal
• Since Version: 3.4
• Assignee: Jordan West
• Reporter: DuyHai Doan

Authors

Jordan West

Reviewers

Pavel Yaskevich

• Created: 18/Mar/16 19:42
• Updated: 16/Apr/19 09:30
• Resolved: 27/Mar/16 22:33

Description

13 bare metal machines

• 6 cores CPU (12 HT)
• 64Gb RAM
• 4 SSD in RAID0

JVM settings:

• G1 GC
• Xms32G, Xmx32G

Data set:

• ≈ 100Gb/per node
• 1.3 Tb cluster-wide
• ≈ 20Gb for all SASI indices

C* settings:

• concurrent_compactors: 1
• compaction_throughput_mb_per_sec: 256
• memtable_heap_space_in_mb: 2048
• memtable_offheap_space_in_mb: 2048

I created 9 SASI indices

• 8 indices with text field, NonTokenizingAnalyser, PREFIX mode, case-insensitive
• 1 index with numeric field, SPARSE mode

After a while, the nodes just gone OOM.

I attach log files. You can see a lot of GC happening while index segments are flush to disk. At some point the node OOM ...

/cc xedin