[CASSANDRA-7220] Nodes hang with 100% CPU load

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce

Environment

C* 2.0.7
4 nodes cluster on 12 core machines

• Severity: Normal
• Assignee: Unassigned
• Reporter: Robert Stupp
• Created: 13/May/14 15:27
• Updated: 16/Apr/19 09:31
• Resolved: 06/Mar/15 17:35

Description

I've ran a test that both reads and writes rows.

After some time, all writes succeeded and all reads stopped.

Two of the four nodes have 16 of 16 threads of the "ReadStage" thread pool running. The number of pending task continuouly grows on these nodes.

I have attached outputs of the stack traces and some diagnostic output from "nodetool tpstats"
"nodetool status" shows all nodes as UN.

I had run that test previously without any issues in with the same configuration.

Some "specials" from cassandra.yaml:

• key_cache_size_in_mb: 1024
• row_cache_size_in_mb: 8192

The nodes running at 100% CPU are "node2" and "node3". node1&node4 are fine.

I'm not sure if it is reproducable - but it's definitly not a good behaviour.

Issue Links

• relates to: CASSANDRA-7705 Safer Resource Management