[CASSANDRA-13923] Flushers blocked due to many SSTables

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Duplicate
• Component/s: Legacy/Local Write-Read Paths, Local/Compaction

Environment

Cassandra 3.11.0
Centos 6 (downgraded JNA)
64GB RAM
12-disk JBOD

• Severity: Normal
• Assignee: Marcus Eriksson
• Reporter: Dan Kinder

Authors

Marcus Eriksson

• Created: 02/Oct/17 21:01
• Updated: 16/Apr/19 09:29
• Resolved: 10/Oct/17 13:39

Description

This started on the mailing list and I'm not 100% sure of the root cause, feel free to re-title if needed.

I just upgraded Cassandra from 2.2.6 to 3.11.0. Within a few hours of serving traffic, thread pools begin to back up and grow pending tasks indefinitely. This happens to multiple different stages (Read, Mutation) and consistently builds pending tasks for MemtablePostFlush and MemtableFlushWriter.

Using jstack shows that there is blocking going on when trying to call getCompactionCandidates, which seems to happen on flush. We have fairly large nodes that have ~15,000 SSTables per node, all LCS.

I seems like this can cause reads to get blocked because they try to acquire a read lock when calling shouldDefragment.

And writes, of course, block once we can't allocate anymore memtables, because flushes are backed up.

We did not have this problem in 2.2.6, so it seems like there is some regression causing it to be incredibly slow trying to do calls like getCompactionCandidates that list out the SSTables.

In our case this causes nodes to build up pending tasks and simply stop responding to requests.

Issue Links

• is duplicated by: CASSANDRA-13215 Cassandra nodes startup time 20x more after upgarding to 3.x