[CASSANDRA-6496] Endless L0 LCS compactions

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.4
• Labels: compaction lcs

Environment

Cassandra 2.0.3, Linux, 6 nodes, 5 disks per node

• Severity: Normal
• Since Version: 2.0 beta 1
• Assignee: Jonathan Ellis Assign to me
• Reporter: Nikolai Grigoriev

Authors

Jonathan Ellis

Reviewers

Marcus Eriksson

• Created: 17/Dec/13 03:00
• Updated: 16/Apr/19 09:31
• Resolved: 17/Dec/13 20:35

Description

I have first described the problem here: http://stackoverflow.com/questions/20589324/cassandra-2-0-3-endless-compactions-with-no-traffic

I think I have really abused my system with the traffic (mix of reads, heavy updates and some deletes). Now after stopping the traffic I see the compactions that are going on endlessly for over 4 days.

For a specific CF I have about 4700 sstable data files right now. The compaction estimates are logged as "[3312, 4, 0, 0, 0, 0, 0, 0, 0]". sstable_size_in_mb=256. 3214 files are about 256Mb (+/1 few megs), other files are smaller or much smaller than that. No sstables are larger than 256Mb. What I observe is that LCS picks 32 sstables from L0 and compacts them into 32 sstables of approximately the same size. So, what my system is doing for last 4 days (no traffic at all) is compacting groups of 32 sstables into groups of 32 sstables without any changes. Seems like a bug to me regardless of what did I do to get the system into this state...