[CASSANDRA-14585] For compactions to progress when pending is greater than 10K cassandra must be continuously restarted and compactions_in_progress purged

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Component/s: Local/Compaction

Environment

Apache Cassandra 2.2.12

• Severity: Normal
• Assignee: Tom Petracca
• Reporter: Zachary Schneider

Authors

Tom Petracca

• Created: 24/Jul/18 07:50
• Updated: 22/Apr/19 14:59

Description

We are experiencing the following behavior when utilizing DTCS on nodes that join the ring and subsequently have many pending compactions.

The symptoms are the following:

nodetool compactionstats reports compactions progressing, however pending count is never significantly reduced.

When investigating we notice the following:

compactions_in_progress is never compacted

find /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/ | wc -l
560730

Eventually we will end up running into to many open files.

In order for compaction to progress we must:

• truncate compactions_in_progress
• restart Cassandra

This compactions will progress rapidly 2K or so and eventually the process will hang again.

In the debug log we see constant flushes to compactions_in_progress.

DEBUG [CompactionExecutor:10] 2018-07-24 07:39:29,552 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 (0%) on-heap, 354 (0%) off-heap INFO [MemtableFlushWriter:9] 2018-07-24 07:39:29,552 Memtable.java:352 - Writing Memtable-compactions_in_progress@1407915569(0.263KiB serialized bytes, 12 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:9] 2018-07-24 07:39:29,553 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8140-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=28982298) DEBUG [CompactionExecutor:16] 2018-07-24 07:39:29,595 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 397 (0%) off-heap INFO [MemtableFlushWriter:13] 2018-07-24 07:39:29,595 Memtable.java:352 - Writing Memtable-compactions_in_progress@535072625(0.298KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:13] 2018-07-24 07:39:29,595 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8141-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=29708684) DEBUG [CompactionExecutor:15] 2018-07-24 07:39:29,619 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 428 (0%) on-heap, 356 (0%) off-heap INFO [MemtableFlushWriter:12] 2018-07-24 07:39:29,619 Memtable.java:352 - Writing Memtable-compactions_in_progress@319301851(0.265KiB serialized bytes, 12 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:12] 2018-07-24 07:39:29,620 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8142-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=30591989) DEBUG [CompactionExecutor:9] 2018-07-24 07:39:29,660 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 396 (0%) off-heap INFO [MemtableFlushWriter:15] 2018-07-24 07:39:29,660 Memtable.java:352 - Writing Memtable-compactions_in_progress@1000370375(0.297KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:15] 2018-07-24 07:39:29,660 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8143-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=31558471) DEBUG [CompactionExecutor:14] 2018-07-24 07:39:29,688 ColumnFamilyStore.java:956 - Enqueuing flush of compactions_in_progress: 460 (0%) on-heap, 389 (0%) off-heap INFO [MemtableFlushWriter:10] 2018-07-24 07:39:29,689 Memtable.java:352 - Writing Memtable-compactions_in_progress@204811984(0.290KiB serialized bytes, 13 ops, 0%/0% of on/off-heap limit) DEBUG [MemtableFlushWriter:10] 2018-07-24 07:39:29,689 Memtable.java:386 - Completed flushing /mnt/data/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/tmp-lb-8144-big-Data.db (0.000KiB) for commitlog position ReplayPosition(segmentId=1532415927699, position=32153206)