[CASSANDRA-4781] Sometimes Cassandra starts compacting system-shema_columns cf repeatedly until the node is killed

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 1.2.0 beta 2

Environment

Ubuntu 12.04, single-node Cassandra cluster

• Severity: Normal
• Assignee: Yuki Morishita
• Reporter: Aleksey Yeschenko

Authors

Yuki Morishita

Reviewers

Sylvain Lebresne

• Created: 09/Oct/12 15:39
• Updated: 16/Apr/19 09:32
• Resolved: 30/Oct/12 16:59

Description

Cassandra starts flushing system-schema_columns cf in a seemingly infinite loop:

INFO [CompactionExecutor:7] 2012-10-09 17:55:46,804 CompactionTask.java (line 239) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32107-Data.db,]. 3,827 to 3,827 (~100% of original) bytes for 3 keys at 0.202762MB/s. Time: 18ms.
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,804 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32107-Data.db')]
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,824 CompactionTask.java (line 239) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32108-Data.db,]. 3,827 to 3,827 (~100% of original) bytes for 3 keys at 0.182486MB/s. Time: 20ms.
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,825 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32108-Data.db')]
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,864 CompactionTask.java (line 239) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32109-Data.db,]. 3,827 to 3,827 (~100% of original) bytes for 3 keys at 0.096045MB/s. Time: 38ms.
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,864 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32109-Data.db')]
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,894 CompactionTask.java (line 239) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32110-Data.db,]. 3,827 to 3,827 (~100% of original) bytes for 3 keys at 0.121657MB/s. Time: 30ms.
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,894 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32110-Data.db')]
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,914 CompactionTask.java (line 239) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32111-Data.db,]. 3,827 to 3,827 (~100% of original) bytes for 3 keys at 0.202762MB/s. Time: 18ms.
INFO [CompactionExecutor:7] 2012-10-09 17:55:46,914 CompactionTask.java (line 119) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-32111-Data.db')]
.........

Don't know what's causing it. Don't know a way to predictably trigger this behaviour. It just happens sometimes.