Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-7220

Nodes hang with 100% CPU load

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Cannot Reproduce
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      C* 2.0.7
      4 nodes cluster on 12 core machines

    • Severity:
      Normal

      Description

      I've ran a test that both reads and writes rows.

      After some time, all writes succeeded and all reads stopped.

      Two of the four nodes have 16 of 16 threads of the "ReadStage" thread pool running. The number of pending task continuouly grows on these nodes.

      I have attached outputs of the stack traces and some diagnostic output from "nodetool tpstats"
      "nodetool status" shows all nodes as UN.

      I had run that test previously without any issues in with the same configuration.

      Some "specials" from cassandra.yaml:

      • key_cache_size_in_mb: 1024
      • row_cache_size_in_mb: 8192

      The nodes running at 100% CPU are "node2" and "node3". node1&node4 are fine.

      I'm not sure if it is reproducable - but it's definitly not a good behaviour.

        Attachments

        1. c-12-read-100perc-cpu.zip
          46 kB
          Robert Stupp
        2. system.log
          4.82 MB
          Robert Rudduck

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                snazy Robert Stupp
              • Votes:
                2 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: