Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14840

Bootstrap of new node fails with OOM in a large cluster

    XMLWordPrintableJSON

    Details

    • Severity:
      Critical
    • Since Version:

      Description

      We are seeing new node addition fails with OOM during bootstrap in a cluster of more than 80 nodes and 3000 CF without any data in those CFs.

       

      Steps to reproduce:

      1. Launch a 3 node cluster
      2. Create 3000 CF in the cluster
      3. Start adding nodes to the cluster one by one
      4. After adding 75-80 nodes, the new node bootstrap fails with OOM.
      ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2018-10-24 03:26:47,870 JVMStabilityInspector.java:78 - Exiting due to error while processing commit log during initialization.
      java.lang.OutOfMemoryError: Java heap space
       at java.util.regex.Pattern.matcher(Pattern.java:1093) ~[na:1.8.0_151]
       at java.util.Formatter.parse(Formatter.java:2547) ~[na:1.8.0_151]
       at java.util.Formatter.format(Formatter.java:2501) ~[na:1.8.0_151]
       at java.util.Formatter.format(Formatter.java:2455) ~[na:1.8.0_151]
       at java.lang.String.format(String.java:2940) ~[na:1.8.0_151]
       at org.apache.cassandra.db.commitlog.AbstractCommitLogService$1.run(AbstractCommitLogService.java:105) ~[apache-cassandra-2.1.16.jar:2.1.16]
       at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]

      Cassandra Version: 2.1.16

      OS: CentOS7

      num_tokens: 256 on each node.

       

      This behavior is blocking us from adding extra capacity when needed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jaid Jai Bheemsen Rao Dhanwada
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: