[CASSANDRA-6614] 2 hours loop flushing+compacting system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce

Environment

ubuntu 12.04

• Severity: Normal
• Assignee: Ryan McGuire
• Reporter: Cyril Scetbon

Authors

Ryan McGuire

• Created: 24/Jan/14 14:05
• Updated: 16/Apr/19 09:31
• Resolved: 11/Nov/15 16:48

Description

It happens when we upgrade one node to 1.2.13 on a 1.2.2 cluster
see http://pastebin.com/YZKUQLXz
If I grep for only InternalResponseStage logs I get http://pastebin.com/htnXZCiT which always displays same account of ops and serialized/live bytes per column family.
When I upgrade one node from 1.2.2 to 1.2.13, for 2h I get the previous messages with a raise of CPU (as it flushes and compacts continually) on all nodes http://picpaste.com/pics/Screen_Shot_2014-01-24_at_09.18.50-ggcCDVqd.1390587562.png
After that, everything is fine and I can upgrade other nodes without any raise of cpus load. when I start the upgrade, the more nodes I upgrade at the same time (at the beginning), the higher the cpu load is http://picpaste.com/pics/Screen_Shot_2014-01-23_at_17.45.56-I3fdEQ2T.1390587597.png