Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-1014

GC storming, possible memory leak

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Won't Fix
    • 0.7.1
    • None
    • None
    • debian lenny amd64 OpenJDK 64-Bit Server VM (build 1.6.0_0-b11, mixed mode)

    • Normal

    Description

      There appears to be a GC issue due to memory pressure in the 0.6 branch. You can see this by starting the server and performing many inserts. Quickly the jvm will consume most of its heap, and pauses for stop-the-world GC will begin. With verbose GC turned on, this can be observed as follows:

      [GC [ParNew (promotion failed): 79703K->79703K(84544K), 0.0622980 secs][CMS[CMS-concurrent-mark: 3.678/5.031 secs] [Times: user=10.35 sys=4.22, real=5.03 secs]
      (concurrent mode failure): 944529K->492222K(963392K), 2.8264480 secs] 990745K->492222K(1047936K), 2.8890500 secs] [Times: user=2.90 sys=0.04, real=2.90 secs]

      After enough inserts (around 75-100 million) the server will GC storm and then OOM.

      jbellis and I narrowed this down to patch 0001 in CASSANDRA-724. Switching LBQ with ABQ made no difference, however using batch mode instead of periodic for the commitlog does prevent the issue from occurring. The attached screenshot shows the heap usage in jconsole first when the issue is exhibiting, a restart, and then the same amount of inserts when it does not.

      Attachments

        1. 1014-2Gheap.png
          90 kB
          Brandon Williams
        2. 1014-commitlog-v2.tar.gz
          18 kB
          Jonathan Ellis
        3. 1014-table.diff
          0.9 kB
          Jonathan Ellis
        4. 724-0001.png
          65 kB
          Brandon Williams
        5. gc2.png
          52 kB
          Lu Ming

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            brandon.williams Brandon Williams
            Votes:
            2 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment