Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-1177

OutOfMemory on heavy inserts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Urgent
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      SunOS 5.10, x86 32bit, Jave Hotspot Server VM 11.2-b01 mixed mode
      Sun SDK 1.6.0_12-b04

    • Severity:
      Critical

      Description

      We have cluster of 6 Cassandra 0.6.2 nodes running under SunOS (see environment).

      On initial import (using the thrift API) we see some weird behavior of half the cluster. While cas04-06 look fine as you can see from the attached munin graphs, the other 3 nodes kept on GCing (see log file) until they became unreachable and went OOM. (This is also why the stats are so spotty - munin could no longer reach the boxes) We have seen the same behavior on 0.6.2 and 0.6.1. This started after around 100 million inserts.

      Looking at the hprof (which is of course to big to attach) we see lots of ConcurrentSkipListMap$Node's and quite some Column objects. Please see the stats attached.

      This looks similar to https://issues.apache.org/jira/browse/CASSANDRA-1014 but we are not sure it really is the same.

        Attachments

        1. bug report.zip
          336 kB
          Torsten Curdt
        2. commitlog.txt
          0.3 kB
          Alexander Simmerl
        3. data.txt
          127 kB
          Alexander Simmerl

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tcurdt Torsten Curdt
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: