Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Fix Version/s: 0.7.1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      debian lenny amd64 OpenJDK 64-Bit Server VM (build 1.6.0_0-b11, mixed mode)

      Description

      There appears to be a GC issue due to memory pressure in the 0.6 branch. You can see this by starting the server and performing many inserts. Quickly the jvm will consume most of its heap, and pauses for stop-the-world GC will begin. With verbose GC turned on, this can be observed as follows:

      [GC [ParNew (promotion failed): 79703K->79703K(84544K), 0.0622980 secs][CMS[CMS-concurrent-mark: 3.678/5.031 secs] [Times: user=10.35 sys=4.22, real=5.03 secs]
      (concurrent mode failure): 944529K->492222K(963392K), 2.8264480 secs] 990745K->492222K(1047936K), 2.8890500 secs] [Times: user=2.90 sys=0.04, real=2.90 secs]

      After enough inserts (around 75-100 million) the server will GC storm and then OOM.

      jbellis and I narrowed this down to patch 0001 in CASSANDRA-724. Switching LBQ with ABQ made no difference, however using batch mode instead of periodic for the commitlog does prevent the issue from occurring. The attached screenshot shows the heap usage in jconsole first when the issue is exhibiting, a restart, and then the same amount of inserts when it does not.

      1. gc2.png
        52 kB
        Lu Ming
      2. 1014-2Gheap.png
        90 kB
        Brandon Williams
      3. 1014-commitlog-v2.tar.gz
        18 kB
        Jonathan Ellis
      4. 1014-table.diff
        0.9 kB
        Jonathan Ellis
      5. 724-0001.png
        65 kB
        Brandon Williams

        Issue Links

          Activity

          Brandon Williams created issue -
          Brandon Williams made changes -
          Field Original Value New Value
          Attachment 724-0001.png [ 12442575 ]
          Brandon Williams made changes -
          Description There appears to be a GC issue due to memory pressure in the 0.6 branch. You can see this by starting the server and performing many inserts. Quickly the jvm will consume most of its heap, and pauses for stop-the-world GC will begin. With verbose GC turned on, this can be observed as follows:

          [GC [ParNew (promotion failed): 79703K->79703K(84544K), 0.0622980 secs][CMS[CMS-concurrent-mark: 3.678/5.031 secs] [Times: user=10.35 sys=4.22, real=5.03 secs]
           (concurrent mode failure): 944529K->492222K(963392K), 2.8264480 secs] 990745K->492222K(1047936K), 2.8890500 secs] [Times: user=2.90 sys=0.04, real=2.90 secs]

          After enough inserts (around 75-100 million) the server will GC storm and then OOM.

          jbellis and I narrowed this down to patch 0001 in CASSANDRA-724. Switching LBQ with ABQ made no difference, however using batch mode instead of periodic for the commitlog does prevent the issue from occurring. The attached screenshot show the heap usage in jconsole first when the issue is exhibiting, a restart, and then the same amount of inserts when it does not.
          There appears to be a GC issue due to memory pressure in the 0.6 branch. You can see this by starting the server and performing many inserts. Quickly the jvm will consume most of its heap, and pauses for stop-the-world GC will begin. With verbose GC turned on, this can be observed as follows:

          [GC [ParNew (promotion failed): 79703K->79703K(84544K), 0.0622980 secs][CMS[CMS-concurrent-mark: 3.678/5.031 secs] [Times: user=10.35 sys=4.22, real=5.03 secs]
           (concurrent mode failure): 944529K->492222K(963392K), 2.8264480 secs] 990745K->492222K(1047936K), 2.8890500 secs] [Times: user=2.90 sys=0.04, real=2.90 secs]

          After enough inserts (around 75-100 million) the server will GC storm and then OOM.

          jbellis and I narrowed this down to patch 0001 in CASSANDRA-724. Switching LBQ with ABQ made no difference, however using batch mode instead of periodic for the commitlog does prevent the issue from occurring. The attached screenshot shows the heap usage in jconsole first when the issue is exhibiting, a restart, and then the same amount of inserts when it does not.
          Jonathan Ellis made changes -
          Attachment 1014.txt [ 12442585 ]
          Jonathan Ellis made changes -
          Fix Version/s 0.6.2 [ 12314931 ]
          Affects Version/s 0.6 [ 12314361 ]
          Affects Version/s 0.6.1 [ 12314867 ]
          Component/s Core [ 12312978 ]
          Jonathan Ellis made changes -
          Attachment 1014-table.diff [ 12442588 ]
          Jonathan Ellis made changes -
          Attachment 1014-commitlog.tar.gz [ 12442589 ]
          Jonathan Ellis made changes -
          Attachment 1014-commitlog-v2.tar.gz [ 12442617 ]
          Jonathan Ellis made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Assignee Jonathan Ellis [ jbellis ]
          Brandon Williams made changes -
          Attachment 1014-2Gheap.png [ 12442749 ]
          Jonathan Ellis made changes -
          Attachment 1014.txt [ 12442585 ]
          Jonathan Ellis made changes -
          Attachment 1014-commitlog.tar.gz [ 12442589 ]
          Jonathan Ellis made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Eric Evans made changes -
          Fix Version/s 0.6.3 [ 12315056 ]
          Fix Version/s 0.6.2 [ 12314931 ]
          Torsten Curdt made changes -
          Link This issue is related to CASSANDRA-1177 [ CASSANDRA-1177 ]
          Lu Ming made changes -
          Attachment gc2.png [ 12448014 ]
          Jonathan Ellis made changes -
          Fix Version/s 0.6.4 [ 12315173 ]
          Fix Version/s 0.6.3 [ 12315056 ]
          Jonathan Ellis made changes -
          Assignee Jonathan Ellis [ jbellis ]
          Fix Version/s 0.7.0 [ 12315212 ]
          Fix Version/s 0.6.4 [ 12315173 ]
          Jonathan Ellis made changes -
          Fix Version/s 0.7.1 [ 12315199 ]
          Fix Version/s 0.7.0 [ 12315212 ]
          Jonathan Ellis made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Gavin made changes -
          Workflow no-reopen-closed, patch-avail [ 12509117 ] patch-available, re-open possible [ 12752210 ]
          Gavin made changes -
          Workflow patch-available, re-open possible [ 12752210 ] reopen-resolved, no closed status, patch-avail, testing [ 12758191 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Brandon Williams
            • Votes:
              2 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development