[CASSANDRA-8549] Not able to bring cassandra cluster to stable state

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate

Environment

centos 6

• Severity: Normal
• Assignee: Unassigned
• Reporter: Rahul Bhardwaj
• Created: 31/Dec/14 08:03
• Updated: 16/Apr/19 09:31
• Resolved: 05/Jan/15 14:34

Description

Hi,

We are using cassandra 2.1 on prod environment (cluster with three 64 GB RAM machines) where a batch of write operations are done per minute (around 10000 inserts) using PHP PDO.

Couple of days back we found our cluster unstable then we restarted cassandra service on all machines, but since then cassandra is busy in flushing bulk of sstable_activity which causes cassandra shutdown or unresponsive due to full heap space (java heap space 8GB).

After several attempts of restarting cassandra service two nodes started. Out of these two nodes, one node's cassandra daemon consumed whole RAM (64 GB). Top command shows 75G of virtual memory consumed and 69 G of Resident memory consumption. Other node is ok.

But node1 again busy in flushing sstable_activity and not getting statrted.

please help in solving this, these issues are on prod environment.

Issue Links

• duplicates: CASSANDRA-8018 Cassandra seems to insert twice in custom PerColumnSecondaryIndex