[CASSANDRA-13980] Compaction deadlock

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Duplicate
• Component/s: Local/Compaction

Environment

Cassandra 3.11.2
48 nodes cluster using LCS, JBOD
Big nodes with many SSTables

• Severity: Critical
• Assignee: Unassigned
• Reporter: Loic Lambiel
• Created: 29/Oct/17 09:05
• Updated: 16/Apr/19 09:29
• Resolved: 30/Oct/17 07:39

Description

While upgrading the cluster from 2.1.16 from 3.11.2, after a few hours most of the upgraded nodes started to go in a compaction infinite loop and showing many events like the one below (always for the same SSTable):

INFO  [CompactionExecutor:4] 2017-10-29 00:28:31,480 LeveledManifest.java:474 - Adding high-level (L5) BigTableReader(path='/var/lib/cassandra/data/datadisk4/blobstore/block-1d63273065b911e49cd7ef0972cffde6/blobstore-block-ka-201694-Data.db') to candidates

Since the log get spammed at a huge rate, I'm unable to get any previous events.

Tried restarts and sstablescrub -m without success. The only workaround that seems to work (so far) was sstablelevelreset.

I've attached the dump.

Issue Links

• is duplicated by: CASSANDRA-13948 Reload compaction strategies when JBOD disk boundary changes