Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8285

Move all hints related tasks to hints private executor

Agile BoardAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.12, 2.1.3
    • None
    • None
    • Cassandra 2.0.11 + java-driver 2.0.8-SNAPSHOT
      Cassandra 2.0.11 + ruby-driver 1.0-beta

    • Normal

    Description

      We ran drivers 3-days endurance tests against Cassandra 2.0.11 and C* crashed with an OOME. This happened both with ruby-driver 1.0-beta and java-driver 2.0.8-snapshot.

      Attached are :

      OOME_node_system.log The system.log of one Cassandra node that crashed
      gc.log.gz The GC log on the same node
      heap-usage-after-gc.png The heap occupancy evolution after every GC cycle
      heap-usage-after-gc-zoom.png A focus on when things start to go wrong

      Workload :
      Our test executes 5 CQL statements (select, insert, select, delete, select) for a given unique id, during 3 days, using multiple threads. There is not change in the workload during the test.

      Symptoms :
      In the attached log, it seems something starts in Cassandra between 2014-11-06 10:29:22 and 2014-11-06 10:45:32. This causes an allocation that fills the heap. We eventually get stuck in a Full GC storm and get an OOME in the logs.

      I have run the java-driver tests against Cassandra 1.2.19 and 2.1.1. The error does not occur. It seems specific to 2.0.11.

      Attachments

        1. 8285.txt
          9 kB
          Aleksey Yeschenko
        2. 8285-v2.txt
          10 kB
          Aleksey Yeschenko
        3. 8285-v3.txt
          13 kB
          Sam Tunnicliffe
        4. gc.log.gz
          5.15 MB
          Pierre Laporte
        5. gc-1416849312.log.gz
          2.23 MB
          Pierre Laporte
        6. heap-usage-after-gc.png
          384 kB
          Pierre Laporte
        7. heap-usage-after-gc-zoom.png
          119 kB
          Pierre Laporte
        8. OOME_node_system.log
          6.01 MB
          Pierre Laporte
        9. system.log.gz
          150 kB
          Pierre Laporte

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            aleksey Aleksey Yeschenko Assign to me
            pingtimeout Pierre Laporte
            Aleksey Yeschenko
            Sam Tunnicliffe
            Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment