Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-8285

Move all hints related tasks to hints private executor



    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.12, 2.1.3
    • None
    • None
    • Cassandra 2.0.11 + java-driver 2.0.8-SNAPSHOT
      Cassandra 2.0.11 + ruby-driver 1.0-beta

    • Normal


      We ran drivers 3-days endurance tests against Cassandra 2.0.11 and C* crashed with an OOME. This happened both with ruby-driver 1.0-beta and java-driver 2.0.8-snapshot.

      Attached are :

      OOME_node_system.log The system.log of one Cassandra node that crashed
      gc.log.gz The GC log on the same node
      heap-usage-after-gc.png The heap occupancy evolution after every GC cycle
      heap-usage-after-gc-zoom.png A focus on when things start to go wrong

      Workload :
      Our test executes 5 CQL statements (select, insert, select, delete, select) for a given unique id, during 3 days, using multiple threads. There is not change in the workload during the test.

      Symptoms :
      In the attached log, it seems something starts in Cassandra between 2014-11-06 10:29:22 and 2014-11-06 10:45:32. This causes an allocation that fills the heap. We eventually get stuck in a Full GC storm and get an OOME in the logs.

      I have run the java-driver tests against Cassandra 1.2.19 and 2.1.1. The error does not occur. It seems specific to 2.0.11.


        1. gc.log.gz
          5.15 MB
          Pierre Laporte
        2. heap-usage-after-gc-zoom.png
          119 kB
          Pierre Laporte
        3. heap-usage-after-gc.png
          384 kB
          Pierre Laporte
        4. OOME_node_system.log
          6.01 MB
          Pierre Laporte
        5. gc-1416849312.log.gz
          2.23 MB
          Pierre Laporte
        6. system.log.gz
          150 kB
          Pierre Laporte
        7. 8285.txt
          9 kB
          Aleksey Yeschenko
        8. 8285-v2.txt
          10 kB
          Aleksey Yeschenko
        9. 8285-v3.txt
          13 kB
          Sam Tunnicliffe

        Issue Links



              aleksey Aleksey Yeschenko
              pingtimeout Pierre Laporte
              Aleksey Yeschenko
              Sam Tunnicliffe
              0 Vote for this issue
              15 Start watching this issue