[CASSANDRA-14239] OutOfMemoryError when bootstrapping with less than 100GB RAM

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Feature/Materialized Views

Environment

Details of the bootstrapping Node

• ProLiant BL460c G7
• 56GB RAM
• 2x 146GB 10K HDD (One dedicated for Commitlog, one for Data, Hints and saved_caches)
• CentOS 7.4 on SD-Card
• /tmp and /var/log on tmpfs
• Oracle JDK 1.8.0_151
• Cassandra 3.11.1

Cluster

• 10 existing Nodes (Up and Normal)

• Severity: Normal
• Since Version: 3.11.1
• Assignee: Unassigned
• Reporter: Jürgen Albersdorfer
• Created: 16/Feb/18 08:12
• Updated: 16/Apr/19 09:29

Description

Hi, I face an issue when bootstrapping a Node having less than 100GB RAM on our 10 Node C* 3.11.1 Cluster.

During bootstrap, when I watch the cassandra.log I observe a growth in JVM Heap Old Gen which gets not significantly freed up any more.

I know that JVM collects on Old Gen only when really needed. I can see collections, but there is always a remainder which seems to grow forever without ever getting freed.

After the Node successfully Joined the Cluster, I can remove the extra RAM I have given it for bootstrapping without any further effect.

It feels like Cassandra will not forget about every single byte streamed over the Network over time during bootstrapping, - which would be a memory leak and a major problem, too.

I was able to produce a HeapDumpOnOutOfMemoryError from a 56GB Node (40 GB assigned JVM Heap). YourKit Profiler shows huge amount of Memory allocated for org.apache.cassandra.db.Memtable (22 GB) org.apache.cassandra.db.rows.BufferCell (19 GB) and java.nio.HeapByteBuffer (11 GB)

Issue Links

• relates to: CASSANDRA-13299 Potential OOMs and lock contention in write path streams