[CASSANDRA-11025] Too many compactions on certain node when too many empty tables are created

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Fix Version/s: 2.0.17
• Component/s: Local/Compaction

Environment

4 nodes with 24 core cpu and 32G ram, CentOS 6.5. Each node is configured with 8G heap size.

• Severity: Normal
• Assignee: Unassigned
• Reporter: Shuo Chen
• Created: 18/Jan/16 02:36
• Updated: 16/Apr/19 09:30
• Resolved: 19/Jan/16 18:10

Description

I have configured 4 nodes of cassandra cluster version 2.0.16. Each node has about 10G load. One of the node has too many pending compactions not too long and too many full gc after it restarts.

Here is part of gc histogram:
num #instances #bytes class name
----------------------------------------------
1: 67758530 2168272960 java.util.concurrent.FutureTask
2: 67759745 1626233880 java.util.concurrent.Executors$RunnableAdapter
3: 67758576 1626205824 java.util.concurrent.LinkedBlockingQueue$Node
4: 67758529 1626204696 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
5: 16935 72995576 [B
6: 240534 11545632 java.nio.HeapByteBuffer
7: 37416 5969800 [C
8: 41447 5624856 <constMethodKlass>
9: 41447 5315504 <methodKlass>
10: 104850 5032800 edu.stanford.ppl.concurrent.SnapTreeMap$Node
11: 4110 4564144 <constantPoolKlass>
12: 104781 3352992 org.apache.cassandra.db.Column
13: 4110 2824016 <instanceKlassKlass>

Here is the nodetool stats:

[cassandra@whaty181 apache-cassandra-2.0.16]$ bin/nodetool compactionstats
pending tasks: 64642341
Active compaction remaining time : n/a

However in system.log, there are not too much log of compaction. I used inotify to monitor the data directory events. There are few events when pending tasks accumulate.

I used jmap to dump the heap and analyzed the java.util.concurrent.FutureTask. It contains many task of CompactionExecutor. I checked the cf of CompactionExecutor. However most of the cfs are never inserted any data since created.

I have created 6 keyspaces and 5 keyspaces never inserted any data. In these 5 keyspaces, 2 keyspaces contain 42 cfs each and 3 keyspaces contains 6 cfs in total. So there are about 90 empty cfs.

All of these cfs are super column families created using casandra-cli

After I dropped these 5 keyspaces and restarted this node, the compaction status of this node is normal again. So I suggests there is many some bug concerning compaction on empty columnfamily.

Issue Links

• duplicates: CASSANDRA-9662 compactionManager reporting wrong pendingtasks