[CASSANDRA-2058] Load spikes due to MessagingService-generated garbage collection

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 0.6.11, 0.7.1

Environment

OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
Ubuntu 8.10
Linux pmc01 2.6.27-22-xen #1 SMP Fri Feb 20 23:58:13 UTC 2009 x86_64 GNU/Linux

• Severity: Normal
• Assignee: Jonathan Ellis
• Reporter: David King

Authors

Jonathan Ellis

Reviewers

Brandon Williams

• Created: 26/Jan/11 01:20
• Updated: 16/Apr/19 09:33
• Resolved: 15/Feb/11 19:48

Description

(Filing as a placeholder bug as I gather information.)

At ~10p 24 Jan, I upgraded our 20-node cluster from 0.6.8->0.6.10, turned on the DES, and moved some CFs from one KS into another (drain whole cluster, take it down, move files, change schema, put it back up). Since then, I've had four storms whereby a node's load will shoot to 700+ (400% CPU on a 4-cpu machine) and become totally unresponsive. After a moment or two like that, its neighbour dies too, and the failure cascades around the ring. Unfortunately because of the high load I'm not able to get into the machine to pull a thread dump to see wtf it's doing as it happens.

I've also had an issue where a single node spikes up to high load, but recovers. This may or may not be the same issue from which the nodes don't recover as above, but both are new behaviour