[CASSANDRA-10829] cleanup + repair generates a lot of logs

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.13, 2.2.5, 3.0.3, 3.3
• Component/s: Legacy/Streaming and Messaging

Environment

5 nodes on Cassandra 2.1.11 (on Debian)

• Severity: Normal
• Since Version: 2.1.8
• Assignee: Marcus Eriksson
• Reporter: Fabien Rousseau

Authors

Marcus Eriksson

Reviewers

Carl Yeksigian

• Created: 08/Dec/15 09:37
• Updated: 16/Apr/19 09:30
• Resolved: 19/Jan/16 08:11

Description

One of our node generates a lot of cassandra logs (int the 10 MB/s) and CPU usage has increased (by a factor 2-3).
This was most probably triggered by a "nodetool snapshot" while a cleanup was already running on this node.

An example of those logs:
2015-12-08 09:15:17,794 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to capture released readers [...]
2015-12-08 09:15:17,794 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to capture all readers [...]
2015-12-08 09:15:17,795 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1923 Spinning trying to capture released readers [...]
2015-12-08 09:15:17,795 INFO [ValidationExecutor:689]ColumnFamilyStore.java:1924 Spinning trying to capture all readers [...]
(I removed SSTableReader information because it's rather long... I can share it privately if needed)
Note that the date has not been changed (only 1ms between logs)

It should not generate that gigantic amount of logs

This is probably linked to: https://issues.apache.org/jira/browse/CASSANDRA-9637