[CASSANDRA-14840] Bootstrap of new node fails with OOM in a large cluster

• Type: Bug
• Status: Open
• Priority: Urgent
• Resolution: Unresolved
• Component/s: Legacy/Streaming and Messaging
• Labels: remove-reopen
• Severity: Critical
• Since Version: 2.1.16
• Assignee: Unassigned
• Reporter: Jai Bheemsen Rao Dhanwada
• Created: 24/Oct/18 05:15
• Updated: 16/Apr/19 09:29

Description

We are seeing new node addition fails with OOM during bootstrap in a cluster of more than 80 nodes and 3000 CF without any data in those CFs.

 

Steps to reproduce:

1. Launch a 3 node cluster
2. Create 3000 CF in the cluster
3. Start adding nodes to the cluster one by one
4. After adding 75-80 nodes, the new node bootstrap fails with OOM.

ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2018-10-24 03:26:47,870 JVMStabilityInspector.java:78 - Exiting due to error while processing commit log during initialization.
java.lang.OutOfMemoryError: Java heap space
 at java.util.regex.Pattern.matcher(Pattern.java:1093) ~[na:1.8.0_151]
 at java.util.Formatter.parse(Formatter.java:2547) ~[na:1.8.0_151]
 at java.util.Formatter.format(Formatter.java:2501) ~[na:1.8.0_151]
 at java.util.Formatter.format(Formatter.java:2455) ~[na:1.8.0_151]
 at java.lang.String.format(String.java:2940) ~[na:1.8.0_151]
 at org.apache.cassandra.db.commitlog.AbstractCommitLogService$1.run(AbstractCommitLogService.java:105) ~[apache-cassandra-2.1.16.jar:2.1.16]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

Cassandra Version: 2.1.16

OS: CentOS7

num_tokens: 256 on each node.

 

This behavior is blocking us from adding extra capacity when needed.

Issue Links

• duplicates: CASSANDRA-11748 Schema version mismatch may leads to Casandra OOM at bootstrap during a rolling upgrade process