Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10449

OOM on bootstrap after long GC pause

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Duplicate
    • 2.1.x
    • None
    • Ubuntu 14.04, AWS

    • Normal

    Description

      I have a 20-node cluster (i2.4xlarge) with vnodes (default of 256) and 500-700GB per node. SSTable counts are <10 per table. I am attempting to provision additional nodes, but bootstrapping OOMs every time after about 10 hours with a sudden long GC pause:

      INFO  [Service Thread] 2015-10-05 23:33:33,373 GCInspector.java:252 - G1 Old Generation GC in 1586126ms.  G1 Old Gen: 49213756976 -> 49072277176;
      ...
      ERROR [MemtableFlushWriter:454] 2015-10-05 23:33:33,380 CassandraDaemon.java:223 - Exception in thread Thread[MemtableFlushWriter:454,5,main]
      java.lang.OutOfMemoryError: Java heap space
      

      I have tried increasing max heap to 48G just to get through the bootstrap, to no avail.

      Attachments

        1. threads.txt
          83 kB
          Mikhail Stepura
        2. thread_dump.log
          221 kB
          Robbie Strickland
        3. system.log.10-05
          18.89 MB
          Robbie Strickland
        4. heap_dump.png
          103 kB
          Robbie Strickland
        5. GCpath.txt
          13 kB
          Mikhail Stepura

        Issue Links

          Activity

            People

              Unassigned Unassigned
              rstrickland Robbie Strickland
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: